How to reduce unnecessary high memory usage in a Databricks cluster?

@Senad Hadzikic - Thanks for the question and using MS Q&A platform.

Based on the provided information, it seems that the cluster is not getting back to the idle state even after the script execution is completed. To reduce unnecessary high memory usage in a Databricks cluster, you can try the following steps:

• Turn on Auto Optimize by adding the following properties to your Spark configuration:

    spark.databricks.delta.optimizeWrite.enabled true spark.databricks.delta.autoCompact.enabled true
  

• Configure your cluster depending on your integration and scaling needs. For cluster configuration details, see Configure clusters.
• You can also try setting the spark.dynamicAllocation.enabled property to true to enable dynamic allocation of executors. This will allow the cluster to automatically scale up or down based on the workload.
• Additionally, you can try setting the spark.memory.fraction property to a lower value, such as 0.4 or 0.5, to reduce the amount of memory allocated to Spark.
• You can also try setting the spark.dynamicAllocation.minExecutors property to a lower value, such as 1, to reduce the number of executors that are running when the cluster is idle.

If these steps do not help, you may need to further investigate the root cause of the high memory usage. You can use the Databricks monitoring and logging features to identify the source of the memory usage and take appropriate actions to optimize your cluster.

Hope this helps. Do let us know if you any further queries.

Hello and thank you for the answer, I'll try that out.
The monitoring tools didn't help much in debugging the issue to be honest. I've done some of my own investigations:

1. Ran this notebook:

%sh
free -h
top -b -n 1 | head -n 20
jps -l

The result:

               total        used        free      shared  buff/cache   available
Mem:            57Gi        35Gi        12Gi       0.0Ki       9.5Gi        21Gi
Swap:            9Gi       0.0Ki         9Gi
top - 11:45:55 up 15 min,  0 users,  load average: 1.38, 1.65, 1.44
Tasks:  32 total,   1 running,  30 sleeping,   0 stopped,   1 zombie
%Cpu(s):  8.2 us,  0.4 sy,  0.0 ni, 91.4 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  58770.0 total,  12420.7 free,  36636.6 used,   9712.7 buff/cache
MiB Swap:  10240.0 total,  10240.0 free,      0.0 used.  22133.4 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
    759 root      20   0   57.3g  33.3g 186028 S 126.7  58.0  17:50.24 java
    365 root      20   0 8775452   1.0g  47792 S   6.7   1.8   1:06.52 java
      1 root      20   0   17964  10776   8272 S   0.0   0.0   0:00.19 systemd
     45 root      20   0   39572  11828  10740 S   0.0   0.0   0:00.05 systemd+
     55 systemd+  20   0   25540  12980   8788 S   0.0   0.0   0:00.01 systemd+
     57 root      20   0    3888   2600   2360 S   0.0   0.0   0:00.00 cron
     58 message+  20   0    8592   4776   4192 S   0.0   0.0   0:00.05 dbus-da+
     61 root      20   0   35280  22048  10344 S   0.0   0.0   0:00.16 network+
     63 syslog    20   0  160960   5056   4496 S   0.0   0.0   0:00.01 rsyslogd
     67 root      20   0   15360   7616   6608 S   0.0   0.0   0:00.02 systemd+
     72 root      20   0   15436   8808   7256 S   0.0   0.0   0:00.00 sshd
     74 root      20   0    2820   1108   1020 S   0.0   0.0   0:00.00 agetty
     76 root      20   0  121200  23328  12720 S   0.0   0.0   0:00.14 unatten+
759 com.databricks.backend.daemon.driver.DriverDaemon
508 org.apache.spark.deploy.master.Master
365 com.databricks.spark.chauffeur.ChauffeurDaemon
5598 sun.tools.jps.Jps

2. Then this notebook:

%python
import os


pid = os.popen("jps | grep DriverDaemon | awk '{print $1}'").read().strip()
heap_dump_command = f"jmap -dump:live,format=b,file=/tmp/heapdump_{pid}.hprof {pid}"
print(os.popen(heap_dump_command).read())


gc_stats_command = f"jstat -gc {pid}"
print(os.popen(gc_stats_command).read())


memory_summary_command = f"jmap -heap {pid}"
print(os.popen(memory_summary_command).read())


open_files_command = "lsof | grep java | wc -l"
print(os.popen(open_files_command).read())

The result:

1.
Dumping heap to /tmp/heapdump_759.hprof ...
Heap dump file created

2.
 S0C    S1C    S0U    S1U      EC       EU        OC         OU       MC     MU    CCSC   CCSU   YGC     YGCT    FGC    FGCT     GCT   
 0.0    0.0    0.0    0.0   29671424.0 73728.0  17424384.0  566112.3  499948.0 474495.4  0.0    0.0       76    4.449   1      0.958    5.406

3.
Attaching to process ID 759, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.392-b08

using thread-local object allocation.
Garbage-First (G1) GC with 13 thread(s)

Heap Configuration:
   MinHeapFreeRatio         = 40
   MaxHeapFreeRatio         = 70
   MaxHeapSize              = 48226107392 (45992.0MB)
   NewSize                  = 1363144 (1.2999954223632812MB)
   MaxNewSize               = 28932308992 (27592.0MB)
   OldSize                  = 5452592 (5.1999969482421875MB)
   NewRatio                 = 2
   SurvivorRatio            = 8
   MetaspaceSize            = 21807104 (20.796875MB)
   CompressedClassSpaceSize = 1073741824 (1024.0MB)
   MaxMetaspaceSize         = 17592186044415 MB
   G1HeapRegionSize         = 8388608 (8.0MB)

Heap Usage:
G1 Heap:
   regions  = 5749
   capacity = 48226107392 (45992.0MB)
   used     = 1418559840 (1352.8440856933594MB)
   free     = 46807547552 (44639.15591430664MB)
   2.9414769648925017% used
G1 Young Generation:
Eden Space:
   regions  = 100
   capacity = 30383538176 (28976.0MB)
   used     = 838860800 (800.0MB)
   free     = 29544677376 (28176.0MB)
   2.7609055770292654% used
Survivor Space:
   regions  = 0
   capacity = 0 (0.0MB)
   used     = 0 (0.0MB)
   free     = 0 (0.0MB)
   0.0% used
G1 Old Generation:
   regions  = 70
   capacity = 17842569216 (17016.0MB)
   used     = 579699040 (552.8440856933594MB)
   free     = 17262870176 (16463.15591430664MB)
   3.2489661829652055% used

91848 interned Strings occupying 12598088 bytes.

4.
lsof: WARNING: can't stat() fuse file system /local_disk0/.wsfs
      Output information may be incomplete.

If this helps you give even better feedback, it didn't help me though.

Thank you for the answer. I applied the suggested changes.
The memory usage is still really huge:

I ran this notebook to get little more details:

%sh
free -h
top -b -n 1 | head -n 20
jps -l

The result:

               total        used        free      shared  buff/cache   available
Mem:            57Gi        35Gi        10Gi       0.0Ki        10Gi        21Gi
Swap:            9Gi       0.0Ki         9Gi
top - 12:54:50 up 32 min,  0 users,  load average: 2.48, 1.54, 1.23
Tasks:  30 total,   1 running,  28 sleeping,   0 stopped,   1 zombie
%Cpu(s):  6.5 us,  0.0 sy,  0.0 ni, 93.5 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  58770.0 total,  11114.9 free,  36754.9 used,  10900.2 buff/cache
MiB Swap:  10240.0 total,  10240.0 free,      0.0 used.  22015.1 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
    796 root      20   0   57.2g  33.7g 185796 S 100.0  58.8  26:35.27 java
    389 root      20   0 8704676   1.0g  47760 S   6.2   1.8   1:27.04 java
      1 root      20   0   17972  10808   8284 S   0.0   0.0   0:00.23 systemd
     41 root      20   0   39576  11764  10668 S   0.0   0.0   0:00.06 systemd+
     71 systemd+  20   0   25540  12948   8756 S   0.0   0.0   0:00.01 systemd+
     77 root      20   0    3888   2396   2160 S   0.0   0.0   0:00.00 cron
     78 message+  20   0    8592   4764   4180 S   0.0   0.0   0:00.04 dbus-da+
     81 root      20   0   35228  21864  10208 S   0.0   0.0   0:00.16 network+
     83 syslog    20   0  160960   4924   4364 S   0.0   0.0   0:00.01 rsyslogd
     87 root      20   0   15512   7772   6780 S   0.0   0.0   0:00.03 systemd+
     92 root      20   0   15436   8880   7332 S   0.0   0.0   0:00.00 sshd
     94 root      20   0    2820   1072    984 S   0.0   0.0   0:00.00 agetty
     99 root      20   0  121080  22996  12532 S   0.0   0.0   0:00.12 unatten+
6881 sun.tools.jps.Jps
532 org.apache.spark.deploy.master.Master
389 com.databricks.spark.chauffeur.ChauffeurDaemon
796 com.databricks.backend.daemon.driver.DriverDaemon

This is happening on both of our clusters that we use.

@Senad Hadzikic - Based on the output you provided, it looks like your cluster is still using a significant amount of memory even after applying the suggested changes. Here are a few more things you can try:

• Check if there are any long-running Spark jobs or notebooks that might be consuming memory. You can check this by going to the "Jobs" and "Notebooks" tabs in the Databricks workspace and looking for any jobs or notebooks that have been running for a long time.
• Try reducing the spark.executor.memory value to allocate less memory to each executor. You can also try reducing the spark.driver.memory value to allocate less memory to the driver.
• Check if there are any external libraries or dependencies that might be causing memory leaks. You can try removing any unnecessary libraries or dependencies and see if that helps reduce memory usage.
• Try monitoring the cluster's memory usage over time using the Databricks monitoring tools. This can help you identify any patterns or trends in memory usage and adjust your settings accordingly.
• Finally, you can try optimizing your Spark code to reduce memory usage. This can include techniques such as using smaller data partitions, using more efficient data serialization formats, and optimizing your Spark configuration settings.

I hope this helps! Let me know if you have any further questions.

Thank you for the response once again. I have optimized the code to the max, as much as possible. These tips helped a bit but we still have a really huge memory usage. I was wondering if there's an additional possible way to release the cached memory which is 22GB (that is not restarting the cluster itself)? I've tried a couple of steps through a notebook cell but still no luck.

And this is the OS-level cache. I just want to be able to clear it, but can't manage to do it with a notebook it seems. Any other way?