Q:

Hadoop on Azure, can I use a different Blob Storage container for I/O?

I am currently working on a project to create a Big Data Architecture on Azure. In order to understand how Azure works, I created a Data Factory and Blob Storage, and set up a pipeline for a word count Hadoop process on an On-Demand HDInsight System.
This is the JSON file for the pipeline:
{
 "name": "MRSamplePipeline5",
    "properties": {
        "description": "Sample Pipeline to Run the Word Count Program",
        "activities": [
            {
                "type": "HDInsightMapReduce",
                "typeProperties": {
                    "className": "wordcount",
                    "jarFilePath": "executables/hadoop-example.jar",
                    "jarLinkedService": "AzureStorageLinkedService",
                    "arguments": [
                        "/davinci.txt",
                        "/WordCountOutput1"
                    ]
                },
                "outputs": [
                    {
                        "name": "MROutput4"
                    }
                ],
                "policy": {
                    "timeout": "01:00:00",
                    "concurrency": 1,
                    "retry": 3
                },
                "scheduler": {
                    "frequency": "Minute",
                    "interval": 15
                },
                "name": "MRActivity",
                "linkedServiceName": "HDInsightOnDemandLinkedService"
            }
        ],
        "start": "2017-07-24T00:00:00Z",
        "end": "2017-07-24T00:00:00Z",
        "isPaused": false,
        "hubName": "testazuredatafact_hub",
        "pipelineMode": "OneTime",
        "expirationTime": "3.00:00:00"
    }
}

It does work, even though the output is a file called "WordCountOutput1/part-r-00000".
My question is: How do I define the input file (davinci.txt) and the output file (Output1) to be in a different container (e.g. "exampledata") of my blob storage?

A:

Hadoop file paths may be specified in a full URI syntax, including scheme and authority, to point at different kinds of file systems (e.g. HDFS vs. Azure vs. S3) and in your specific case, different Azure storage containers.  The relevant scheme for Azure Storage access is "wasb".  The authority contains the container and account.  For example, consider the following hadoop fs -ls commands.
# WASB backed by container "test" in Azure Storage account "cnauroth"
hadoop fs -ls wasb://test@cnauroth.blob.core.windows.net/users/cnauroth

# WASB backed by container "qa" in Azure Storage account "cnauroth"
hadoop fs -ls wasb://qa@cnauroth.blob.core.windows.net/users/cnauroth

# WASB backed by container "production" in Azure Storage account "cnauroth-live"
hadoop fs -ls wasb://production@cnauroth-live.blob.core.windows.net/users/cnauroth

Each command, executed from the same client host, lists a different Azure Storage account/container.
You can use the same URI syntax when passing arguments to your job submissions.

