GCLOUD_DATAPROC_JOBS_SUBMIT_SPARK(1)

NAME

gcloud dataproc jobs submit spark - submit a Spark job to a cluster

SYNOPSIS

gcloud dataproc jobs submit spark --cluster=CLUSTER [--archives=[ARCHIVE,...]] [--async] [--bucket=BUCKET] [--driver-log-levels=[PACKAGE=LEVEL,...]] [--files=[FILE,...]] [--jars=[JAR,...]] [--labels=[KEY=VALUE,...]] [--properties=[PROPERTY=VALUE,...]] [--region=REGION] [--class=MAIN_CLASS] [--jar=MAIN_JAR] [GCLOUD_WIDE_FLAG ...] [-- JOB_ARGS ...]

DESCRIPTION
Submit a Spark job to a cluster.

POSITIONAL ARGUMENTS

    [-- JOB_ARGS ...]
        The arguments to pass to the driver. The '--' argument must be specified between gcloud specific args on the left and JOB_ARGS on the right.

REQUIRED FLAGS

    --cluster=CLUSTER
        The Dataproc cluster to submit the job to.

OPTIONAL FLAGS

    --archives=[ARCHIVE,...]
        Comma separated list of archives to be provided to the job. must be one of the following file formats: .zip, .tar, .tar.gz, or .tgz.

    --async
        Does not wait for the job to run.

    --bucket=BUCKET
        The Cloud Storage bucket to stage files in. Defaults to the cluster's configured bucket.

    --driver-log-levels=[PACKAGE=LEVEL,...]
        A list of package to log4j log level pairs to configure driver logging. For example: root=FATAL,com.example=INFO

    --files=[FILE,...]
        Comma separated list of files to be provided to the job.

    --jars=[JAR,...]
        Comma separated list of jar files to be provided to the executor and driver classpaths.

    --labels=[KEY=VALUE,...]
        A list of label KEY=VALUE pairs to add.

    --properties=[PROPERTY=VALUE,...]
        A list of key value pairs to configure Spark.

    --region=REGION
        Specifies the Cloud Dataproc region to use. Each Cloud Dataproc region constitutes an independent resource namespace constrained to deploying instances into Google Compute Engine zones inside the region. The default value of "global" is a special multi-region namespace which is capable of deploying instances into all Google Compute Engine zones globally, and is disjoint from other Cloud Dataproc regions. Overrides the default dataproc/region property value for this command invocation.

    --class=MAIN_CLASS
        The class containing the main method of the driver. Must be in a provided jar or jar that is already on the classpath

    --jar=MAIN_JAR
        The HCFS URI of jar file containing the driver jar.

GCLOUD WIDE FLAGS
These flags are available to all commands: --account, --configuration, --flatten, --format, --help, --log-http, --project, --quiet, --trace-token, --user-output-enabled, --verbosity. Run $ gcloud help for details.

EXAMPLES
To submit a Spark job that runs the main class of a jar, run:
    $ gcloud dataproc jobs submit spark --cluster my_cluster \
        --jar my_jar.jar arg1 arg2
To submit a Spark job that runs a specific class of a jar, run:
    $ gcloud dataproc jobs submit spark --cluster my_cluster \
        --class org.my.main.Class --jars my_jar1.jar,my_jar2.jar arg1 \
        arg2
To submit a Spark job that runs a jar that is already on the cluster, run:
    $ gcloud dataproc jobs submit spark --cluster my_cluster \
        --class org.apache.spark.examples.SparkPi \
        --jars file:///usr/lib/spark/lib/spark-examples.jar 1000