MINI-PROJECT-REPORT
CS 836: Software Production Engineering
~Mayank Jain(IMT2019053)
DevOps
DevOps involves a collection of methodologies that integrate software development (Dev) and IT operations (Ops) in order to optimize the entire software development lifecycle. Its primary objective is to foster collaboration and communication between development and operations teams, which enables swift and efficient development, deployment, and maintenance of software applications. Key practices in DevOps encompass continuous integration and delivery, automation, infrastructure as code, monitoring, and feedback loops, among others.
DevOps involves a combination of practices, tools, and cultural norms, including:
Infrastructure as Code (IaC): By treating infrastructure as software code, organizations can utilize the same tools and processes for managing both their infrastructure and application code.
Continuous Integration and Continuous Delivery/Deployment (CI/CD): The process entails the automation of software development tasks such as building, testing, and deployment, with the aim of ensuring that changes can be delivered rapidly and dependably.
Collaboration and communication: This requires dismantling barriers between teams and encouraging communication and cooperation among various stakeholders, including development, operations, and others.
1.1 Why DevOps?
DevOps is a software development and IT operations methodology that fosters collaboration to enhance the efficiency of the software application delivery process. Its benefits to organizations include:
Enhanced speed to market: DevOps enables faster and more frequent software releases by removing traditional silos between development and operations teams, allowing organizations to quickly adapt to changing customer demands and market conditions.
Improved collaboration: DevOps promotes better communication, collaboration, and knowledge sharing between development and operations teams, which results in more effective collaboration.
Better quality: By facilitating continuous integration and delivery, DevOps allows for faster and more frequent testing, reducing the risk of defects and improving the quality of software releases.
Increased efficiency: DevOps automates repetitive tasks and employs standardized processes, reducing manual errors and time required to complete tasks, resulting in greater efficiency.
Enhanced customer experience: DevOps enables organizations to deliver software that meets customer needs and expectations quickly and reliably, resulting in a better customer experience.
In conclusion, DevOps has significant advantages for organizations in improving collaboration, speed, quality, efficiency, and customer satisfaction.
- Tools used :
A calculator is a program that conducts essential mathematical calculations, including addition, subtraction, multiplication, and division. Our version incorporates additional functions like calculating square roots, factorials, natural logarithms, and exponential powers. A DevOps toolchain can be employed to develop this program, which involves a series of procedures:
Version Control: Use a version control system like Git and GitHub to keep track of changes in the codebase.
Maven: It is a build automation tool primarily used for Java projects. It manages project dependencies, compiles source code, and generates distributable artifacts such as JARs and WARs.
Test: One effective way to automate the testing process is by using a testing framework such as JUnit. With the help of this framework, it is possible to conduct unit tests and produce reports on the coverage and outcomes of these tests.
Continuous integration: Employing a CI/CD server such as Jenkins facilitates the constant integration of your code.
Webhooks: To utilize webhooks, it is necessary to convert the private IP address of the local machine into a public IP address using ngrok.
Containerize: Create a container using a containerization tool such as Docker, and upload it to Docker Hub.
Logging: Use log4j, a Java-based logging utility that offers a versatile logging framework for Java applications. It is utilized for logging messages generated by applications, enabling developers to monitor the execution of their code and troubleshoot issues that might arise during runtime.
Monitoring: Employ a monitoring tool such as the ELK Stack to oversee the application and produce notifications for errors and malfunctions.
Deploy: Utilize a deployment tool such as Ansible for the purpose of conducting configuration management and executing deployment.
2.1 Development
Java 8 is the programming language utilized for constructing the codebase. The calculator's diverse functionalities are implemented using a switch case, with individual functions (like – power, logarithm, square root and factorial) allocated to each feature, depending on the selected option.
2.2 Testing
JUnit is utilized to write and execute unit tests in Java for the calculator's functionalities. Unit testing is a software testing approach that scrutinizes individual units or components of an application to verify their accuracy. The test cases, comprising true and false positives, are included in the test file, which is used to assess the code during the project development process. JUnit is a testing framework specifically designed for Java, aiding developers in writing and running unit tests effectively.
2.3 Logging
Log4j, which stands for Log for Java, is a well-liked logging utility based on Java that offers a versatile and customizable logging infrastructure for Java applications. With Log4j, programmers can produce log statements within their code to record real-time data regarding the behavior, speed, and issues of the application.
Log4j provides support for log levels, allowing developers to control the granularity of the information captured in the logs. Log levels include DEBUG, INFO, WARN, ERROR, and FATAL.
- Project Build (Maven Build)
Maven is mainly utilized as a build automation tool for Java-based projects, assisting in the organization of the build procedure by specifying project architecture, dependencies, and plugins.
Some of the key applications of Maven are:
Build automation: Maven automates the process of building and packaging software, including compiling source code, running tests, and creating distributable artifacts.
Dependency management: Maven handles project dependencies, including downloading required libraries and managing version conflicts.
Project structure: Maven provides a standard project structure that helps developers organize their code and resources in a consistent way.
Convention over configuration: Maven uses conventions and default settings to reduce the need for configuration and make the build process simpler and more predictable.
Plugin system: Maven has a powerful plugin system that allows developers to extend and customize the build process, adding new functionality as needed.
Overall, Maven simplifies the build process for Java projects, improves code quality, and makes it easier for teams to collaborate. To use certain dependencies like JUnit in our project, we define the pom.xml file, a part of the Maven build.
By executing the command mvn clean install, the codebase will be built, dependencies will be fetched, test cases will be run and reported. In the current directory, a new directory called target will be generated which will have the executable .jar files.
3.1 Source Code Management – Using Git and GitHub
Source code management (SCM), alternatively referred to as version control, involves the process of monitoring and handling alterations made to software code as time progresses. By employing SCM, developers can work together on a project, keep a record of changes, and regulate access to different iterations of the code.
Git is a distributed version control system used for tracking changes in source code during software development. It was created by Linus Torvalds in 2005. Git allows multiple developers to work on the same codebase simultaneously, and provides features for managing conflicts and merging changes. It has become one of the most widely used version control systems in the software industry. Git is open source and can be used for any type of project, from small personal projects to large enterprise software development.
GitHub is a web-based platform for hosting and managing software development projects that use Git for version control. It provides features such as issue tracking, pull requests, and code reviews, which facilitate collaboration among developers. GitHub also offers social networking features such as follower and following lists, and the ability to star and fork repositories. It is widely used by individuals, open source projects, and enterprises for software development and sharing. GitHub is owned by Microsoft and offers both free and paid subscription plans.
The development work can be started locally and the directory can be added to git using git init, followed by git add .
Creation of GitHub repository
The repository can be created on GitHub, as shown below:
The repository can be found at this URL: https://github.com/Mayankjain909/SPE-mini-project/tree/main/MINI_PROJECT-main
Ngrok
Ngrok is a software that empowers developers to make a web server on their local machine available on the internet. It constructs a secure pathway between the local computer and a public URL, which can be accessed from any location. This enables the testing and distribution of applications without necessitating a server that is publicly accessible.
While ngrok is active, it generates a transient public URL that grants access to the local server. Sharing this URL with others allows them to view and engage with the in-progress application. The ngrok tunnel incorporates HTTPS encryption, which guarantees that traffic between the local machine and the public URL is secure. During the development and testing stages, developers can employ ngrok to obtain webhooks from external services.
To initiate ngrok, execute the ensuing command: ./ngrok http . Subsequently, substitute with the actual port number of the local web server that we desire to expose. ngrok will then generate a public URL that enables access to the local web server from any location. To discover the URL, look in the ngrok terminal window, beneath the "Forwarding" section. Copy the URL and paste it into a web browser for testing purposes. At this point, the local web server is accessible on the internet via the ngrok public URL. Remember that terminating the ngrok process or closing the terminal window will end the ngrok session.
Webhooks
Webhooks are a way for web applications to receive real-time updates from other services or applications. They send a notification when a particular event occurs, such as a new user sign-up or a transaction completion. Webhooks enable developers to automate processes and integrate systems.
To set up GitHub webhooks in Jenkins, follow these steps:
First, ensure that you have installed the GitHub plugin in Jenkins.
In GitHub, navigate to the repository where you want to set up webhooks and go to "Settings" > "Webhooks" > "Add webhook".
In the "Payload URL" field, enter the URL for your Jenkins server followed by "/github-webhook/".
Select the events that should trigger the webhook, such as push or pull request events.
Save the webhook configuration and test it by making a change in the repository that should trigger the webhook.
In Jenkins, create a new job or edit an existing one, and under "Build Triggers", select "GitHub hook trigger for GITScm polling".
Save the job configuration and trigger a build to test the webhook integration.
GitHub webhooks in Jenkins can be used to automate various aspects of the development workflow, such as building and testing code, deploying applications, and notifying team members of new events. By using webhooks, we can streamline the development process and reduce manual effort.
Jenkins offers various types of build triggers that automate the initiation of a build process:
• SCM Polling: This trigger periodically checks the source code repository for updates and commences a build if any changes are detected.
• Schedule: This trigger starts a build at a specified time or on a recurrent basis.
• Build after other projects: This trigger starts a build once one or more specified projects have been built successfully.
• Remote trigger: This trigger enables remote programs or scripts to commence a build through a URL or API call.
• Periodic build: This trigger begins a build at set intervals, such as every day or hour.
• GitHub hook trigger for GITScm polling: This trigger permits Jenkins to receive incoming webhooks from GitHub and start a build when alterations are made in the repository.
• Pipeline trigger: This trigger allows one pipeline to trigger another pipeline. These triggers can be set up in the Jenkins project configuration, and multiple triggers can be employed together to start a build based on different circumstances.
Jenkins
Jenkins is an open-source automation platform based on Java, equipped with Continuous Integration (CI) plugins. It is employed to generate and test software projects periodically, which simplifies the process of incorporating modifications for developers and enables users to obtain a new build. In this project, the Jenkins pipeline was utilized to manage the process up to delivery, i.e., continuous delivery. To access the Jenkins service, type the URL "http://localhost:8080" into the address bar of a web browser.
Jenkins can be installed by following these steps:
wget -q -O - https://pkg.jenkins.io/debian-stable/jenkins.io.key | sudo apt-key add -
sudo sh -c ’echo deb http://pkg.jenkins.io/debian-stable binary/ > /etc/apt/sources.list
sudo apt install ca-certificates
sudo apt-get update
sudo apt-get install jenkins
6.1 Plugins
Plugins are a key feature of Jenkins that allow developers to extend the functionality of the software and customize it to meet their specific needs. Jenkins has a vast library of over 1,500 plugins, with new plugins being developed and added to the library on a regular basis. The plugins which we need for this project are the ones for Git, Docker, Ansible and Maven. Plugins can be installed from the Plugin Manager under Manage Jenkins.
6.2 Global tool configuration
Under the Global Tool Configuration options in Manage Jenkins, we have to specify the configuration for Maven, Git and Ansible. Though they are almost everytime automatically populated, it is a good idea to have a check.
6.3 Manage Credentials
Under the Manage Credentials options in Manage Jenkins, we have to specify the login credentials for Docker Hub and GitHub.
6.4 Jenkins Pipeline
Jenkins Pipeline is a method for automating the necessary steps involved in building, testing, and deploying a software application. To define the pipeline, one usually creates a Jenkinsfile, which contains the specifications, stages, and configurations required for executing a continuous delivery pipeline. Jenkins pipelines can be administered either through the Jenkins web interface or directly through the Jenkinsfile, providing automated and version-controlled pipeline management.
The pipeline script consists of various stages, which can be customized based on the needs of the application.
Git Pull: Pull the codebase from a remote repository hosted on GitHub.
Maven Build: It creates a jar file with all of our source code’s dependencies in it. A new target folder with the new jar file will be created when the original target folder with the previous dependencies is destroyed.
Create Docker Image: On our local system, it is used to create images that are subsequently uploaded to our Docker Hub, allowing us to fetch the image and execute the application on other systems.
Publish Docker Image: We are uploading the image to our Docker Hub in this stage so that anyone may download it. To grant permissions, we must execute the command sudo chmod 666 /var/run/docker.sock on the localhost.
Clean Docker Image: In this step, we remove the Docker image from our local system.
Ansible Deploy: Fetching the image from the Docker Hub and running it on the hosts specified in the inventory file.
6.5 Jenkins Run
- Ansible
Ansible is an open-source automation tool that helps in configuring and managing computer systems. It is designed to be simple, agentless, and powerful, allowing users to automate repetitive tasks, such as software installations, configuration management, and application deployment. Ansible fetches the Docker image from the Docker Hub and deploys it across several machines.
7.1 Docker File
A Dockerfile is a text document that outlines a series of instructions used to create a Docker image. These instructions typically include details on the base image to use, any additional packages or software required, configuration settings, and any other actions necessary to prepare the image.
Using a Dockerfile provides a way for developers to automate the process of building Docker images, which ensures consistency across various environments and platforms. Additionally, Dockerfiles can be version-controlled and shared among team members, simplifying collaboration on creating and deploying applications in a Docker environment.
Once a Dockerfile is created, it can be utilized to generate a Docker image using the "docker build" command. The resulting image can then be used to execute containers that contain the specified software and configurations.
The following are some of the main fieds of DockerFile:
• FROM: Specifies the base image that the new image will be built upon.
• RUN: Runs a command inside the container while building the image.
• COPY or ADD: Copies files or directories from the host system into the container.
• WORKDIR: Sets the working directory for any RUN, CMD, ENTRYPOINT, COPY, and ADD instructions that follow.
• EXPOSE: Informs Docker that the container will listen on the specified network ports at runtime.
• CMD or ENTRYPOINT: Specifies the command to be executed when the container starts.
• ENV: Sets environment variables that will be available in the container
7.2 DockerHub
The created image is pushed to the Docker Hub. Docker Hub is a cloud-based repository that allows developers to store and share their Docker container images with other developers or users. It is essentially a central hub for Docker images, where developers can find, download, and share pre-built Docker images that can be used to run their applications in containers.
7.3 Inventory
In Ansible, an inventory file is a configuration file that specifies the hosts and groups of hosts that Ansible will manage. The inventory file contains a list of hostnames or IP addresses that Ansible will connect to and execute tasks on.
7.4 PlayBook
In Ansible, a playbook is a file or set of files that defines a set of tasks to be executed on one or more hosts. Playbooks are written in YAML format and are used to automate complex tasks such as configuration management, application deployment, and orchestration.
- Continuous Monitoring
The ELK stack is a set of open-source software tools that are commonly used for log management and analysis. ELK is an acronym that stands for Elasticsearch, Logstash, and Kibana.
Elasticsearch is a distributed search and analytics engine that is designed to provide fast and efficient full-text search capabilities. It is used to store and index large amounts of data in realtime, and provides advanced querying capabilities.
Logstash is a data processing pipeline that is used to collect, process, and transform data from various sources. It is often used to collect log data from different sources and parse it into a structured format that can be easily analyzed. Kibana is a data visualization tool that is used to create interactive dashboards and visualizations from data stored in Elasticsearch.
It allows users to easily explore and analyze data using a web-based interface, and provides a range of visualization options, including charts, graphs, and maps. Together, the ELK stack provides a powerful and flexible platform for log management and analysis, and is widely used in a variety of industries, including IT, security, and business intelligence.