Can not create a Path from an empty string

Question

Can not create a Path from an empty string

UDAYA SRINIVASARAO KOTHAMASU 1

Can not create a Path from an empty string
sshuser@hn0-dhl-hd:~$ hdfs dfs -ls -h wasbs://******@dhlsghdinsightprodsa2.blob.core.windows.net/RMGR_RP_20260814*.csv.gz
-ls: Can not create a Path from an empty string
Usage: hadoop fs [generic options]
        [-appendToFile <localsrc> ... <dst>]
        [-cat [-ignoreCrc] <src> ...]
        [-checksum [-v] <src> ...]
        [-chgrp [-R] GROUP PATH...]
        [-chmod [-R] <MODE[,MODE]... | OCTALMODE> PATH...]
        [-chown [-R] [OWNER][:[GROUP]] PATH...]
        [-concat <target path> <src path> <src path> ...]
        [-copyFromLocal [-f] [-p] [-l] [-d] [-t <thread count>] [-q <thread pool queue size>] <localsrc> ... <dst>]
        [-copyToLocal [-f] [-p] [-crc] [-ignoreCrc] [-t <thread count>] [-q <thread pool queue size>] <src> ... <localdst>]
        [-count [-q] [-h] [-v] [-t [<storage type>]] [-u] [-x] [-e] [-s] <path> ...]
        [-cp [-f] [-p | -p[topax]] [-d] [-t <thread count>] [-q <thread pool queue size>] <src> ... <dst>]
        [-createSnapshot <snapshotDir> [<snapshotName>]]
        [-deleteSnapshot <snapshotDir> <snapshotName>]
        [-df [-h] [<path> ...]]
        [-du [-s] [-h] [-v] [-x] <path> ...]
        [-expunge [-immediate] [-fs <path>]]
        [-find <path> ... <expression> ...]
        [-get [-f] [-p] [-crc] [-ignoreCrc] [-t <thread count>] [-q <thread pool queue size>] <src> ... <localdst>]
        [-getfacl [-R] <path>]
        [-getfattr [-R] {-n name | -d} [-e en] <path>]
        [-getmerge [-nl] [-skip-empty-file] <src> <localdst>]
        [-head <file>]
        [-help [cmd ...]]
        [-ls [-C] [-d] [-h] [-q] [-R] [-t] [-S] [-r] [-u] [-e] [<path> ...]]
        [-mkdir [-p] <path> ...]
        [-moveFromLocal [-f] [-p] [-l] [-d] <localsrc> ... <dst>]
        [-moveToLocal <src> <localdst>]
        [-mv <src> ... <dst>]
        [-put [-f] [-p] [-l] [-d] [-t <thread count>] [-q <thread pool queue size>] <localsrc> ... <dst>]
        [-renameSnapshot <snapshotDir> <oldName> <newName>]
        [-rm [-f] [-r|-R] [-skipTrash] [-safely] <src> ...]
        [-rmdir [--ignore-fail-on-non-empty] <dir> ...]
        [-setfacl [-R] [{-b|-k} {-m|-x <acl_spec>} <path>]|[--set <acl_spec> <path>]]
        [-setfattr {-n name [-v value] | -x name} <path>]
        [-setrep [-R] [-w] <rep> <path> ...]
        [-stat [format] <path> ...]
        [-tail [-f] [-s <sleep interval>] <file>]
        [-test -[defswrz] <path>]
        [-text [-ignoreCrc] <src> ...]
        [-touch [-a] [-m] [-t TIMESTAMP (yyyyMMdd:HHmmss) ] [-c] <path> ...]
        [-touchz <path> ...]
        [-truncate [-w] <length> <path> ...]
        [-usage [cmd ...]]

Generic options supported are:
-conf <configuration file>        specify an application configuration file
-D <property=value>               define a value for a given property
-fs <file:///|hdfs://namenode:port> specify default filesystem URL to use, overrides 'fs.defaultFS' property from configurations.
-jt <local|resourcemanager:port>  specify a ResourceManager
-files <file1,...>                specify a comma-separated list of files to be copied to the map reduce cluster
-libjars <jar1,...>               specify a comma-separated list of jar files to be included in the classpath
-archives <archive1,...>          specify a comma-separated list of archives to be unarchived on the compute machines

The general command line syntax is:
command [genericOptions] [commandOptions]

Usage: hadoop fs [generic options] -ls [-C] [-d] [-h] [-q] [-R] [-t] [-S] [-r] [-u] [-e] [<path> ...]
n empty string

Peram Venkateswara Reddy 260 Reputation points Microsoft External Staff Moderator

2026-08-14T08:48:12.6566667+00:00
Hi @UDAYA SRINIVASARAO KOTHAMASU ,
This is a shell quoting issue, not a storage or permissions problem.

What's happening

Look at what Hadoop printed — the error plus the full Usage: banner. Hadoop prints that when it receives an empty string as the path argument. Your storage account and container never came into it.

The cause is the unquoted *. Bash treats RMGR_RP_20260814*.csv.gz as a glob too, and tries to expand it against the local filesystem of the head node before hdfs is invoked. There's no matching local file, so the argument is either mangled or (if nullglob is set in your shell) removed entirely — leaving hdfs dfs -ls -h with nothing, which is exactly the empty-string error you got.

The fix — quote the path

hdfs dfs -ls -h 'wasbs://<container>@dhlsghdinsightprodsa2.blob.core.windows.net/RMGR_RP_20260814*.csv.gz'

Single quotes stop bash from touching the *, so Hadoop's own globber receives it and matches server-side. You can also escape it (\*) or set -o noglob for the session, but quoting is cleanest.

How to confirm this is what happened

Run the same command prefixed with echo — no quotes:

echo hdfs dfs -ls -h wasbs://<container>@dhlsghdinsightprodsa2.blob.core.windows.net/RMGR_RP_20260814*.csv.gz

If the path is missing or altered in the echoed output, bash ate it and the quoting fix applies. Takes two seconds and confirms the diagnosis before you change anything else.

If the quoted version returns "No such file or directory"

That's a different error and means the wildcard reached Hadoop correctly but matched nothing — a genuine path or filename issue. Then check:

# List the container root hdfs dfs -ls 'wasbs://<container>@dhlsghdinsightprodsa2.blob.core.windows.net/' # If files land in a dated folder, list that folder hdfs dfs -ls 'wasbs://<container>@dhlsghdinsightprodsa2.blob.core.windows.net/<folder>/' # Recursive search by name pattern hdfs dfs -ls -R 'wasbs://<container>@dhlsghdinsightprodsa2.blob.core.windows.net/' | grep RMGR_RP_20260814

The grep variant is worth keeping regardless — it's reliable and gives you the exact key names, which matters because blob containers have no real directory hierarchy: the / in a blob name only looks like a path.

Two things worth knowing

If this container is your cluster's default storage, you can use the short form and skip the long URI entirely: hdfs dfs -ls -h '/RMGR_RP_20260814*.csv.gz'. Check with Ambari → HDFS → Configs → fs.defaultFS.

The Azure Storage team has discontinued active development on WASB and recommends the ABFS driver for both Blob and ADLS Gen2. Not urgent for this command, but worth planning for if this is a production pipeline.

References

Use Azure storage with Azure HDInsight clusters — https://learn.microsoft.com/azure/hdinsight/hdinsight-hadoop-use-blob-storage

Azure Storage overview in HDInsight — https://learn.microsoft.com/azure/hdinsight/overview-azure-storage

The Azure Blob Filesystem driver (ABFS) — https://learn.microsoft.com/azure/storage/blobs/data-lake-storage-abfs-driver
UDAYA SRINIVASARAO KOTHAMASU 1 Reputation point

2026-08-14T08:53:05.9766667+00:00

The issue just started yesterday and noticed this morning after failure of the batch. Program is working from last 6 years
UDAYA SRINIVASARAO KOTHAMASU 1 Reputation point

2026-08-15T16:51:45.7933333+00:00

Hi Venkat,

Thanks for attempt the issue. When pyspark batch pulls the file with '' error happening. if you give explicit file it works fine. All the above individual commands with full file qualifier it works fine. Program automaticaly pulls the file and process as part of batch. Suddenly from last two days program fail to fetch the files "".

1 answer

Your answer

UDAYA SRINIVASARAO KOTHAMASU 1 Reputation point

2026-08-14T08:53:05.9766667+00:00

The issue just started yesterday and noticed this morning after failure of the batch. Program is working from last 6 years
UDAYA SRINIVASARAO KOTHAMASU 1 Reputation point

2026-08-15T16:51:45.7933333+00:00

Hi Venkat,

Thanks for attempt the issue. When pyspark batch pulls the file with '' error happening. if you give explicit file it works fine. All the above individual commands with full file qualifier it works fine. Program automaticaly pulls the file and process as part of batch. Suddenly from last two days program fail to fetch the files "".

Answer 1

Hi @UDAYA SRINIVASARAO KOTHAMASU
Thank you for the additional details.

Since you confirmed that:

An explicit filename works successfully.
The same hdfs dfs -ls commands work when the complete filename is provided.
The failure occurs only when the * wildcard is used.
The batch has been working for several years and the issue started only recently.

this helps narrow the issue to how the wildcard path is being handled by the shell/PySpark batch, rather than a general connectivity or storage permission issue.

Could you please try the following from the same environment/user where the batch is running?

hdfs dfs -ls -h 'wasbs://******@dhlsghdinsightprodsa2.blob.core.windows.net/RMGR_RP_20260814*.csv.gz'

Please note the single quotes around the complete path. This prevents the Linux shell from expanding * before the command reaches Hadoop.

Also, please run:

echo hdfs dfs -ls -h wasbs://******@dhlsghdinsightprodsa2.blob.core.windows.net/RMGR_RP_20260814*.csv.gz

and

echo hdfs dfs -ls -h 'wasbs://******@dhlsghdinsightprodsa2.blob.core.windows.net/RMGR_RP_20260814*.csv.gz'

Please share the output of both commands.

Since the failure is occurring from the PySpark batch, could you also share the portion of the PySpark code/job configuration where this wildcard path is constructed and passed to Hadoop/HDFS? Please mask any credentials or sensitive information.

This will help us determine whether the wildcard is being expanded by the shell, PySpark, or the Hadoop filesystem layer.

If possible, please also provide:

The exact error from the failed PySpark batch.
The last successful execution date/time.
The first failed execution date/time.
Whether the same batch is running with the same cluster/image/runtime version as before.
Whether any recent changes were made to the cluster, script, shell environment, Python/PySpark version, or storage configuration.

Once we have these details, we can narrow down why the wildcard behavior changed even though the same batch had been working previously.