Online Root Cause Analysis

Docker Setup

Installation

Check if Docker Compose is installed:

docker-compose -v

If not installed, follow the Docker Getting Started Docs.

Built Image

To build the image:

docker-compose build

Check that the image has been successfully pulled:

docker image list

Example output:

REPOSITORY               TAG       IMAGE ID       CREATED          SIZE
giobbu/kafka-spark-app   latest    <containerId>  About an hour ago  10.6GB

Run the Application

To start the container:

docker-compose run --rm app

• --rm: flag to remove the container after it stops.

Start Zookeper and Kafka services

Logged as root user in usr/bin start kafka server

start_services.sh

Check kafka.log and zookeeper.log are in home/guest

anaconda3  kafka  kafka.log  notebooks  spark  zookeeper.log

Spark Installation

In home/guest

pyspark --version

Output

Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 3.5.6
      /_/
                        
Using Scala version 2.12.18, OpenJDK 64-Bit Server VM, 11.0.25

Start Jupyter Notebook

Switch to guest user

su guest

And start jupyter notebook

notebook

Connect to http://127.0.0.1:8889/tree?token=<token>

Docker Configuration Details (TODO)

Dockerfile

Docker-Compose

• Services
• Volume