MercaData

PySpark jobs for Mercafacil ETL.

Prepare your environment

Validate the Java installation on your system

java -version

Check if the JAVA_HOME environment variable is set on your system

echo $JAVA_HOME

If not, set JAVA_HOME

export JAVA_HOME="/usr/lib/jvm/java-11-openjdk-amd64/"

Install venv:

sudo apt update

sudo apt upgrade

sudo apt install python3.12-venv

Activate the virtual environment and install requirements:

python3.12 -m venv ./venv

source venv/bin/activate

pip install -r requirements.txt

When you're done working with the virtual environment, you can deactivate it by running:

deactivate

Running

cd mercadata/etl

python3 launcher.py gold UpSellCategoria -e prd -m standalone --dry-run

python3 launcher.py bronze Vendas -e prd -m standalone -d 20220102T010203 --noop

python3 launcher.py bronze Vendas -e prd -m standalone -d 20220102T010203