Auto loader

Hi @Vineet S

Greetings & Welcome to Microsoft Q&A forum! Thanks for posting your query!

To automatically add a new column to a Delta table when it is not present, you can enable schema evolution in your write operations. When schema evolution is enabled, any columns that are present in the source data but missing from the target Delta table will be automatically added during the write transaction.

For example, when using Auto Loader or a batch write operation, you can set the mergeSchema option to true. This will ensure that new columns from the source data are appended to the end of the Delta table schema.

The following example demonstrates using the mergeSchema option with a batch write operation:

(spark.read
  .table(source_table)
  .write
  .option("mergeSchema", "true")
  .mode("append")
  .saveAsTable("table_name")
)

For additional information, please refer the below Microsoft documentations:

Update Delta Lake table schema

How does Auto Loader schema evolution work?

I hope this information helps. Please do let us know if you have any further queries.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.