Jobs and batch jobs
Change many rows with SQL as one batch job, or run your own Spark script in the background.
Long work runs in the background as a job, which you follow by ID.
| You want to | Use |
|---|---|
| Change rows with a few SQL statements, published all at once | nexus.start_batch_job() |
| Run your own Python script | nexus.jobs.submit() |
Before you start
- Your key's Access allows writing.
- The has ; otherwise a job is refused before it starts.
- Every example assumes
nexus = Nexus.connect().
Jobs spend credits
A job spends the workspace's credits while it runs. Check your balance in SettingsUsage & limits.
Batch jobs
A batch job runs 1 to 50 statements that change rows and publishes them as one change: all land, or none do.
Start a batch job
import time
from klarety import Nexus, RefusedError, WaitingForOwnerError
with Nexus.connect() as nexus:
try:
job = nexus.start_batch_job([
"UPDATE orders SET status = 'closed' WHERE ordered_at < DATE '2026-01-01'",
"DELETE FROM orders WHERE status = 'test'",
])
except RefusedError as refused:
raise SystemExit(f"Refused: {refused}")
try:
state = nexus.jobs.describe(job["job_id"])
while state["state"] in ("running", "publishing"):
time.sleep(5)
state = nexus.jobs.describe(job["job_id"])
print("Ended as", state["state"])
except WaitingForOwnerError as waiting:
print("Waiting for the owner's OK:", waiting, waiting.job_id)Follow it
jobs.describe() reports running, then publishing, then how the job ended: succeeded, failed, declined or expired.
Waiting for the owner
Some changes need the data owner's OK. Then jobs.describe() raises WaitingForOwnerError with the job_id to follow. The job is held, not failed: never resend it.
Rules
- Each statement must read or change rows. Change columns with Tables and schemas instead.
- An empty table has nothing to start from. Add its first rows with
tables.edit_rows()or a load.
Spark script jobs
jobs.submit() runs a Python Spark script inside the resources your plan allows.
Submit a script
from klarety import Nexus, RefusedError
with Nexus.connect() as nexus:
try:
job = nexus.jobs.submit(script_uri, args=["--day", "2026-11-09"])
except RefusedError as refused:
raise SystemExit(f"Job refused: {refused}")
print(job["job_id"], job["state"])script_uriis a.pyfile in your workspace's own storage; anywhere else is refused.argsare passed to the script, up to 64.- A job runs at most as long as your plan allows one statement to run.
Follow or cancel it
print(nexus.jobs.describe(job["job_id"])) # submitted, running, succeeded, failed or cancelled
nexus.jobs.cancel(job["job_id"])When a job fails, exit_reason says why.
What can go wrong
| Exception | Cause | What to do |
|---|---|---|
RefusedError | A statement does not change rows, the script is not a .py file in your storage, or you asked for more than your plan allows. | Read the message and fix the request. |
InsufficientCreditsError | The workspace cannot pay for the job. | Add credits. |
WaitingForOwnerError | The change waits for the owner's OK. | Follow job_id; do not resend. |
NotFoundError | No such job. | Check the ID. |
Next steps
- Run SQL for reads.
- Spark sessions for interactive Spark.
- Limits and rate limits for your plan's job resources.