PySpark "JAVA_HOME is not set" / Py4JError in CI

By Daniel Zoghalchali·Latchkey

Spark runs on the JVM, so PySpark cannot start a session without Java. When the runner has no JDK on PATH and JAVA_HOME is unset, Spark fails immediately with "JAVA_HOME is not set and no java could be found".

What this error means

Creating a SparkSession fails at startup with "JAVA_HOME is not set and no java could be found in your path" or a Py4JError indicating the JVM could not launch.

JAVA_HOME is not set and no java could be found in your path,
please set the JAVA_HOME variable in your environment to match the
location of your Java installation.

Common causes

No JDK installed on the runner

A slim Python image has no Java, so Spark cannot find a JVM to launch.

JAVA_HOME not exported in the step

Java is installed but JAVA_HOME is unset or points at a wrong path, so Spark cannot locate it.

How to fix it

Install a JDK and set JAVA_HOME

1. Use the setup-java action (or install a JDK package) before running PySpark.
2. It sets JAVA_HOME and puts java on PATH.
3. Then create the SparkSession.

- uses: actions/setup-java@v4
  with:
    distribution: temurin
    java-version: '17'

Export JAVA_HOME explicitly when Java is preinstalled

If a JDK is on the image, point JAVA_HOME at it for the Spark step.

export JAVA_HOME=/usr/lib/jvm/temurin-17-jdk
export PATH="$JAVA_HOME/bin:$PATH"

How to prevent it

• Install a JDK with setup-java before any Spark step.
• Match the Java version to your Spark version requirements.
• Set JAVA_HOME in the job env so every Spark step sees it.

Frequently asked questions

Related guides

References

• Python documentation
• pip documentation
• GitHub Actions documentation