Share via

Spark Batch - Get Spark Batch Jobs

  • Reference
Service:
Synapse
API Version:
2020-12-01

List all spark batch jobs which are running under a particular spark pool.

GET {endpoint}/livyApi/versions/{livyApiVersion}/sparkPools/{sparkPoolName}/batches
With optional parameters: 
GET {endpoint}/livyApi/versions/{livyApiVersion}/sparkPools/{sparkPoolName}/batches?from={from}&size={size}&detailed={detailed}

URI Parameters

Name In Required Type Description
endpoint
 path True

string

The workspace development endpoint, for example https://myworkspace.dev.azuresynapse.net.
livyApiVersion
 path True

string

Valid api-version for the request.
sparkPoolName
 path True

string

Name of the spark pool.
detailed
 query

boolean

Optional query param specifying whether detailed response is returned beyond plain livy.
from
 query

integer

int32

Optional param specifying which index the list should begin from.
size
 query

integer

int32

Optional param specifying the size of the returned list.

        By default it is 20 and that is the maximum.

Responses

Name Type Description
200 OK

SparkBatchJobCollection

Success

Examples

List all spark batch jobs

Sample request

GET myWorkspace.dev.azuresynapse.net/livyApi/versions/2020-12-01/sparkPools/mySparkPool/batches?from=0&size=2&detailed=True

Sample response

Status code:
200 
{
  "from": 0,
  "total": 2,
  "sessions": []
}

Definitions

Name Description
LivyStates

The batch state
PluginCurrentState
SchedulerCurrentState
SparkBatchJob
SparkBatchJobCollection

Response for batch list operation.
SparkBatchJobResultType

The Spark batch job result.
SparkBatchJobState
SparkErrorSource
SparkJobType

The job type.
SparkRequest
SparkScheduler
SparkServiceError
SparkServicePlugin

LivyStates

The batch state

Name Type Description
busy

string

dead

string

error

string

idle

string

killed

string

not_started

string

recovering

string

running

string

shutting_down

string

starting

string

success

string

PluginCurrentState

Name Type Description
Cleanup

string

Ended

string

Monitoring

string

Preparation

string

Queued

string

ResourceAcquisition

string

Submission

string

SchedulerCurrentState

Name Type Description
Ended

string

Queued

string

Scheduled

string

SparkBatchJob

Name Type Description
appId

string

The application id of this session
appInfo

object

The detailed application info.
artifactId

string

The artifact identifier.
errorInfo

SparkServiceError[]

The error information.
id

integer

The session Id.
jobType

SparkJobType

The job type.
livyInfo

SparkBatchJobState

log

string[]

The log lines.
name

string

The batch name.
pluginInfo

SparkServicePlugin

The plugin information.
result

SparkBatchJobResultType

The Spark batch job result.
schedulerInfo

SparkScheduler

The scheduler information.
sparkPoolName

string

The Spark pool name.
state

LivyStates

The batch state
submitterId

string

The submitter identifier.
submitterName

string

The submitter name.
tags

object

The tags.
workspaceName

string

The workspace name.

SparkBatchJobCollection

Response for batch list operation.

Name Type Description
from

integer

The start index of fetched sessions.
sessions

SparkBatchJob[]

Batch list
total

integer

Number of sessions fetched.

SparkBatchJobResultType

The Spark batch job result.

Name Type Description
Cancelled

string

Failed

string

Succeeded

string

Uncertain

string

SparkBatchJobState

Name Type Description
currentState

string

the Spark job state.
deadAt

string

time that at which "dead" livy state was first seen.
jobCreationRequest

SparkRequest

killedAt

string

the time that at which "killed" livy state was first seen.
notStartedAt

string

the time that at which "not_started" livy state was first seen.
recoveringAt

string

the time that at which "recovering" livy state was first seen.
runningAt

string

the time that at which "running" livy state was first seen.
startingAt

string

the time that at which "starting" livy state was first seen.
successAt

string

the time that at which "success" livy state was first seen.

SparkErrorSource

Name Type Description
Dependency

string

System

string

Unknown

string

User

string

SparkJobType

The job type.

Name Type Description
SparkBatch

string

SparkSession

string

SparkRequest

Name Type Description
archives

string[]

args

string[]

className

string

conf

object

driverCores

integer

driverMemory

string

executorCores

integer

executorMemory

string

file

string

files

string[]

jars

string[]

name

string

numExecutors

integer

pyFiles

string[]

SparkScheduler

Name Type Description
cancellationRequestedAt

string

currentState

SchedulerCurrentState

endedAt

string

scheduledAt

string

submittedAt

string

SparkServiceError

Name Type Description
errorCode

string

message

string

source

SparkErrorSource

SparkServicePlugin

Name Type Description
cleanupStartedAt

string

currentState

PluginCurrentState

monitoringStartedAt

string

preparationStartedAt

string

resourceAcquisitionStartedAt

string

submissionStartedAt

string