Skip to content

Commit

Permalink
[SPARK-51155][CORE] Make SparkContext show total runtime after stop…
Browse files Browse the repository at this point in the history
…ping

### What changes were proposed in this pull request?

This PR aims to make `SparkContext` show total runtime after stopping.

### Why are the changes needed?

In Spark UI, we can check `Total Uptime`.
<img width="275" alt="Screenshot 2025-02-10 at 21 41 48" src="https://github.com/user-attachments/assets/d31ede44-ac07-428e-b26d-4c90cfda17d2" />

It would be great if the log shows the total uptime of `SparkContext`.

**BEFORE**
```
$ bin/run-example SparkPi 2>&1 | grep SparkContext | tail -n1
25/02/10 21:41:13 INFO SparkContext: Successfully stopped SparkContext
```

**AFTER**
```
$ bin/run-example SparkPi 2>&1 | grep SparkContext | tail -n1
25/02/10 21:39:34 INFO SparkContext: Successfully stopped SparkContext (Uptime: 3453 ms)
```

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Manually run the job and check the log because this is a log-related PR.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #49878 from dongjoon-hyun/SPARK-51155.

Authored-by: Dongjoon Hyun <dongjoon@apache.org>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
  • Loading branch information
dongjoon-hyun committed Feb 11, 2025
1 parent d3be2a7 commit e7709f0
Showing 1 changed file with 2 additions and 1 deletion.
3 changes: 2 additions & 1 deletion core/src/main/scala/org/apache/spark/SparkContext.scala
Original file line number Diff line number Diff line change
Expand Up @@ -2405,7 +2405,8 @@ class SparkContext(config: SparkConf) extends Logging {
ResourceProfile.clearDefaultProfile()
// Unset YARN mode system env variable, to allow switching between cluster types.
SparkContext.clearActiveContext()
logInfo("Successfully stopped SparkContext")
logInfo(log"Successfully stopped SparkContext (Uptime: " +
log"${MDC(LogKeys.TOTAL_TIME, System.currentTimeMillis() - startTime)} ms)")
}


Expand Down

0 comments on commit e7709f0

Please sign in to comment.