If you are new to the word Azure, then let me tell you Azure
is a cloud. It has multiple applications and services. Azure Data Lake is one
such a service that is provided through Azure. Azure is provided by Microsoft
and we have to create Azure Subscription to start using it. Microsoft provides
free trail subscription of 30 days and after that we have to pay as we use the
cloud Azure.
Coming to our topic of discussion, Azure Data Lake is a
service provided for Big Data where we can securely store data of any size –
any kind and do analysis. Azure Data Lake (will be referred as ADL here after)
has following important parts,
1. Azure Data Lake Store (ADLS)
2. Azure
Data Lake Analytics (ADLS)
3. U-SQL
Azure Data Lake Store
(ADLS): As the name explains, it is a store for data. ADLS is a service and
place where Data of any kind and any size is imported and stored.
ADLS can also have RDBMS objects like Tables, Views, Functions or Stored
Procedures. It is a massive storage where one can store trillions of files of
size more than Petabytes.
Azure Data Lake
Analytics (ADLA): It is a service that lets us do analysis of data that are
stored in ADLS. In order to do Data Analytics or create ADLA service, it is
mandatory to connect ADLA service to ADLS. ADLA uses special kind of structured
query language, unlike the one used for RDMS, called U-SQL. U-SQL scripts are
written in ADLA service and are executed through Jobs. Every U-SQL script we
write is executed through job. We can extract and transform data present in
ADLA and produce output for analysis with ADLA service by running U-SQL jobs.
U-SQL: U-SQL is
combination of SQL and C#. Microsoft has come up with this scripting language by
integrating the features of C# into SQL. This helps user call C# function
directly in SQL. Also, developer can write his/her own C# custom code that can
be used in U-SQL.
U-SQL is used in ADLA to extract data from ADLS, Transform extracted data and generated the transformed data as output. All U-SQL scripts are run from ADLA through job.
