Showing posts with label Power_Query. Show all posts
Showing posts with label Power_Query. Show all posts

Friday, September 5, 2025

How to generate additional columns based on existing columns using M-Query in Power BI

How to generate additional % columns based on existing columns using List. Accumulate () Function in Power Query
Scenario:
Suppose we have a sample data with below fields:
[Year], [Month], [Product], [Gross_Sales], [COGS], [Operating_Expenses], [Interset_Taxes]


Now, based on the [Gross_Sales] column, we need to generate the expense % columns as per below: 
[% COGS], [% Operating_Expenses], [% Interset_Taxes]

We can achieve this Scenario using the following M-Query:
let
    /* Original data source: */
    Source = Table.FromRows(
        Json.Document(
            Binary.Decompress(
                Binary.FromText("hZGxDsIwDET/pXOHxImd5AcYECyIreoQRIdKiCJAfD/xDVVpVbpcpOtd+uw0TUWGTFVX+3wvesrXfihnir6oS1Dv4VRtPaZ33aXoubt1n/7VD1pl1hgRFbUoMk8rx/wsesiP9/DQj05NsohHgTON//KIZ01zxC80LbzBI6xXk1hU4qIy4wFNSqhoMfA6TcClzmBgETgbNDEQVplQZDh/aISA7TGvDXDWedhgVKMdG7BLs8ETsEpnHEbgxQhzHiCTp/HNygra9gs=", BinaryEncoding.Base64),
                Compression.Deflate
            )
        ),
        let _t = ((type nullable text) meta [Serialized.Text = true]) in
        type table [
            Year = _t, Month = _t, Product = _t, Gross_Sales = _t,
            COGS = _t, Operating_Expenses = _t, Interest_Taxes = _t
        ]
    ),

    /* Change column data types: */
    ChangeType = Table.TransformColumnTypes(Source, {
        {"Year", Int64.Type}, {"Month", type text}, {"Product", type text},
        {"Gross_Sales", Int64.Type}, {"COGS", Int64.Type},
        {"Operating_Expenses", Int64.Type}, {"Interest_Taxes", Int64.Type}
    }),

    /* List of columns based on which % columns will be generated: */
    Cols_List = {"COGS", "Operating_Expenses", "Interest_Taxes"},

    /* Generate additional Percentage Columns */
    Add_Ratios = List.Accumulate(
        Cols_List,
        ChangeType,
        (i, c) => Table.AddColumn(
            i,
            "% " & c,
            each if ([Gross_Sales] = null or [Gross_Sales] = 0) then null
                 else (Record.Field( _, c) / [Gross_Sales]),
            type nullable Percentage.Type
        )
    )
in
    Add_Ratios

--------------------------------------------------------------
Notes:
In the above List.Accumulate(i,c) function, the variables will work as per below:
c: The current column name from Cols_List being processed in that iteration.
i: The intermediate table result that accumulates with each added column. It starts as ChangeType and grows with each new % Column.

If Cols_List = {"COGS", "Operating_Expenses"}, then the loop will work as:
First iteration: c = "COGS"
Adds column % COGS to ChangeType, result stored in i

Second iteration: c = "Operating_Expenses"
Adds column % Operating_Expenses to the table from step 1

And so on, until all columns in Cols_List have their corresponding % columns added.

Output:

--------------------------------------------------------------
The above M-Query Code can also be return as per below:
let
    /* Original data source: */
    Source = Table.FromRows(
        Json.Document(
            Binary.Decompress(
                Binary.FromText("hZGxDsIwDET/pXOHxImd5AcYECyIreoQRIdKiCJAfD/xDVVpVbpcpOtd+uw0TUWGTFVX+3wvesrXfihnir6oS1Dv4VRtPaZ33aXoubt1n/7VD1pl1hgRFbUoMk8rx/wsesiP9/DQj05NsohHgTON//KIZ01zxC80LbzBI6xXk1hU4qIy4wFNSqhoMfA6TcClzmBgETgbNDEQVplQZDh/aISA7TGvDXDWedhgVKMdG7BLs8ETsEpnHEbgxQhzHiCTp/HNygra9gs=", BinaryEncoding.Base64),
                Compression.Deflate
            )
        ),
        let _t = ((type nullable text) meta [Serialized.Text = true]) in
        type table [
            Year = _t, Month = _t, Product = _t, Gross_Sales = _t,
            COGS = _t, Operating_Expenses = _t, Interest_Taxes = _t
        ]
    ),

    /* Change column data types: */
    ChangeType = Table.TransformColumnTypes(Source, {
        {"Year", Int64.Type}, {"Month", type text}, {"Product", type text},
        {"Gross_Sales", Int64.Type}, {"COGS", Int64.Type},
        {"Operating_Expenses", Int64.Type}, {"Interest_Taxes", Int64.Type}
    }),

    /* Define a Temp Table with flags to skip or keep columns: */
    tbl_ColumnFlags = #table(
        {"Field_Name", "Skip_Flag"},
        {
            {"Year", true},
            {"Month", true},
            {"Product", true},
            {"Gross_Sales", true},
            {"COGS", false},
            {"Operating_Expenses", false},
            {"Interest_Taxes", false}
        }
    ),

    /* Get list of columns where Skip = false: */
    Cols_List = Table.SelectRows(tbl_ColumnFlags, each [Skip_Flag] = false)[Field_Name],

    /* Generate additional Percentage Columns */
    Add_Ratios = List.Accumulate(
        Cols_List,
        ChangeType,
        (i, c) => Table.AddColumn(
            i,
            "% " & c,
            each if ([Gross_Sales] = null or [Gross_Sales] = 0) then null
                 else (Record.Field( _, c) / [Gross_Sales]),
            type nullable Percentage.Type
        )
    )
in
    Add_Ratios

--------------------------------------------------------------------------------------------------------
Thanks, TAMATAM ; Business Intelligence & Analytics Professional
--------------------------------------------------------------------------------------------------------

Thursday, June 5, 2025

How to load data from a Data Warehouse in Azure Databrics Catalog using Power BI

How to load data from a Data Warehouse in Azure Databrics Catalog with Native SQL Query using Power BI
Unity Catalog:
It is a data governance solution in Databricks that provides centralized access control, auditing, lineage tracking, and data discovery across multiple workspaces. It ensures secure and organized data management.

Catalog:
It is a top-level container within Unity Catalog that holds multiple Databases (Schemas). It helps in data isolation and logical organization.
Example: A SalesData catalog may contain databases like Retail, Wholesale, E-commerce.

Databricks primarily operates on a Lakehouse architecture, which combines elements of data lakes and data warehouses. While Databricks SQL Warehouses are used for querying and analytics, Databricks also supports Databases (Schemas) within Unity Catalog for structured data organization.

SQL Warehouses: 
Compute resources optimized for running SQL queries on structured data. Supports BI tools, Delta Lake integration, and fast query execution.

Databases (Schemas): 
The Database (Schema) exists inside a Catalog and is used for logical grouping of related data. It contains tables, views, functions, and models.
Example: Inside the Retail database, you might store tables like Customers, Orders, and Products.

Notes:
When connecting to Databricks Unity Catalog, Power Query may flatten the hierarchy and display the Catalog as a Database.
This happens because traditional databases (like SQL Server) don’t have a Catalog layer, so Power Query maps the Catalog to a Database for compatibility.

Since Databricks Unity Catalog introduces a three-tier hierarchy (Catalog → Database → Table/View), Power Query may simplify this by treating the Catalog as a Database.
Databricks does not function like a traditional database system (such as SQL Server or PostgreSQL), but it does provide database-like structures within Unity Catalog for managing data efficiently.

--------------------------------------------------------------------------------------------------
Scenario:
To connect and load data from a Data Warehouse in Azure Databrics Catalog, we need the following details like, Server Hostname, HTTP Path, which are mandatory.

I have defined the Parameters in the Power Query to hold the Server Hostname, HTTP Path, Catalog/DB Name, and Schema name as shown below samples (Dummy values).

We will use these parameters in the following examples. Please make sure to pass your own correct values for these parameters while you are testing.

p_ServerHostName =  adb-server_host_id.3.azuredatabricks.net
p_http_Path = sql/protocolv1/o/server_host_id/cluster_id
p_Catalog_DB = MyCatlog_db
p_Schema = MySchema

p_Load_By_Year = 2021This Parameter is used to limit the Data Load from the Query. 

Notes:
In the Azure Databrics Workspace, we can find the Server Hostname and HTTP path values from the following ways:
Compute >[Cluster_Name] > Advanced Options > JDBC/ODBC:
Server hostname: adb-server_host_id.3.azuredatabricks.net
HTTP path: sql/protocolv1/o/server_host_id/cluster_id

SQL Warehouses> [MyWarehouse_Name] > Connection Details:
Server hostname: adb-server_host_id.3.azuredatabricks.net 
HTTP path: /sql/1.0/warehouses/warehouse_id

1) Loading a View or Table from the Data Warehouse in Azure Databrics Catalog:
In Power BI, we use default "Azure Datarbics" connector to connect and load a View or Table from a Data Warehouse in Azure Databrics Catalog.

We need to pass only the Server Hostname and HTTP Path as shown below.


After Connecting to the Source, it will open a dialogue box with available Catalogs and their underlying Warehouses/Databases and Objects to select.

The following is the sample Power Query logic of an object (View) loaded, which is updated with Parameters defined above:

let
    Source = Databricks.Catalogs(p_ServerHostName,p_http_Path, [Catalog=null, Database=null, EnableAutomaticProxyDiscovery=null]),
    Src_Database = Source{[Name=p_Catalog_DB,Kind="Database"]}[Data],
    Src_Schema = Src_Database{[Name=p_Schema,Kind="Schema"]}[Data],
    Src_Object = Src_Schema{[Name="MyViewName",Kind="View"]}[Data]
in
    Src_Object

Note:
If you want to load Table instead of View, change the 
Kind="View" to Kind="Table"

2) Loading data using a Native SQL Query from the Data Warehouse in Azure Databrics Catalog:
To connect and load data by running a Native SQL Query from the Data Warehouse in Azure Databrics Catalog, we need to pass the Server Hostname, HTTP Path and Catalog Name are mandatory.
Please make sure include the Schema.Table/View Name in the Native SQL Query section.

The following is the sample Power Query logic of data loaded using Native SQL Query, which is updated with Parameters defined above:
let
    vCY = Date.Year(DateTime.LocalNow()),
    vYearsList = " IN (" & Number.ToText(vCY) & "," & Number.ToText(vCY-1) & "," & Number.ToText(vCY-2) & ")", 
    vLoadYears = if p_Load_By_Year is null then vYearsList else ">= " & Number.ToText(p_Load_By_Year),
    vSrc_Qry = "SELECT transact_id, transact_date, field3, field4, field5, fieldN
    FROM " & p_Schema & ".fact_Transactions fact
    Where YEAR(fact.transact_date) " & vLoadYears,
    Source = Value.NativeQuery(Databricks.Catalogs(p_ServerHostName, p_http_Path
    [Catalog=p_Catalog_DB, Database=null, EnableAutomaticProxyDiscovery=null]){[Name=p_Catalog_DB,Kind="Database"]}[Data], 
    vSrc_Qry, null, [EnableFolding=true]),
    ChangType = Table.TransformColumnTypes(Source,{{"transact_date", type date}})
in
    ChangType

Notes:
p_Load_By_Year is a Year Parameter is used to limit the Data Load from the Query. If this Parameter is Blank (), then the Data will be loaded by default for last 3 Years (from Current Year). Otherwise, it loads the Loads the data >= 2021 (Year value from Parameter)


--------------------------------------------------------------------------------------------------------
Thanks, TAMATAM ; Business Intelligence & Analytics Professional
--------------------------------------------------------------------------------------------------------

Friday, March 21, 2025

How to generate list of Dates available between Start Date and End Date using Power Query

How to create list of Dates falls between Start Date and End Date using Power Query
Scenario:
Let's say we have two date columns in a Dataset as Task_Start_Date and Task_End_Date as shown below.


Now we need to generate a new Date column as "Task_Active_Dates" based on the no. of Dates falls between Task_Start_Date and Task_End_Date.

For example, Task_Start_Date = 01-Jan-2021 and Task_End_Date= 05-Jan-2021, then there are 5 no. of dates falls between those dates (inclusive of Start and End Dates).

We can achieve this Scenario, using the below Power Query logic:

let
Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("ddDLCQAhDATQXnJ2IYn/WsT+23DZuYwLAS/jk4m6lqg973J1kyRaKZjs9PdGweHOPilkeI76C7xEXuE16m9wvpIphQ7v7PyYAR9R/4Tzll3/o9+Ba6aVa5rsfQA=", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [Task_Start_Date = _t, Task_End_Date = _t, Task_Owner_ID = _t]),
    
ChangeType = Table.TransformColumnTypes(Source,{{"Task_Start_Date", type date}, {"Task_End_Date", type date}, {"Task_Owner_ID", Int64.Type}}),

TaskActiveDates =
Table.AddColumn(ChangeType, "Task_Active_Dates",
    each List.Dates(
    [Task_Start_Date], 
    Duration.Days([Task_End_Date]-[Task_Start_Date])+1,
    #duration(1,0,0,0)
)),
    ExpandDates = Table.ExpandListColumn(TaskActiveDates, "Task_Active_Dates")
in
    ExpandDates

Result:

--------------------------------------------------------------------------------------------------------
Thanks, TAMATAM ; Business Intelligence & Analytics Professional
--------------------------------------------------------------------------------------------------------

Wednesday, March 5, 2025

How to generate Random Sample Dimension and Fact Tables using Power Query

How to generate Random Sample Dimension and Fact Tables using M-Query
Scenario:
Suppose if we want to generate the Dimension and Tables with Random Sample data using Power Query.

1) dim_Date - a sample Calendar dimension with list of Dates for 2021.
2) dim_Customer - a customer dimension with sample list of Customers.
3) dim_Products - a products dimension with sample list of Products
4) fact_Sales_Orders - a fact table with random sample of Order details data.

We can create those sample tables using the following Power Query logics.

1) dim_Date: The sample Calendar dimension with list of Dates for 2021.

let
    /* Generate a list of dates for the entire year 2021 */
    StartDate = #date(2021, 1, 1),
    EndDate = #date(2021, 12, 31),
    DateList = List.Dates(StartDate, Duration.Days(EndDate - StartDate) + 1, #duration(1, 0, 0, 0)),

    /* Convert the list of dates into a table */
    DateTable = Table.FromList(DateList, Splitter.SplitByNothing(), {"Date"}),

    /* Add additional columns for required attributes */
    ExtendedTable = Table.AddColumn(DateTable, "Year", each Date.Year([Date]), Int64.Type),
    AddMonth = Table.AddColumn(ExtendedTable, "Month", each Text.Proper(Text.Start(Date.MonthName([Date]), 3)), type text),
    AddDay = Table.AddColumn(AddMonth, "Day", each Date.Day([Date]), Int64.Type),
    AddMonthYear = Table.AddColumn(AddDay, "Month_Year", each [Month] & "-" & Text.From(Date.Year([Date])), type text),
    AddQuarter = Table.AddColumn(AddMonthYear, "Quarter", each "Q" & Text.From(Date.QuarterOfYear([Date])), type text),
    AddQtrYear = Table.AddColumn(AddQuarter, "Qtr_Year", each [Quarter] & "-" & Text.From(Date.Year([Date])), type text),

    /* Change data types as necessary */
    FinalTable = Table.TransformColumnTypes(AddQtrYear, { {"Date", type date}, {"Year", Int64.Type}, {"Month", type text}, {"Day", Int64.Type}, {"Month_Year", type text}, {"Quarter", type text}, {"Qtr_Year", type text} })
in
    FinalTable

Result:


2) dim_Customer: The customer dimension with sample list of Customers.

let
    CustomerNames = {"Sun Hot", "Moon Cool", "Water Flow", "Sky Star", "Earth Strong", "Fire Blaze", "Ocean Deep", "Cloud Soft", "Wind Swift", "Mountain High"},
    Regions = {"North", "South", "East", "West"},
    Source = Table.FromRecords(
        List.Transform({1..10}, each [
            Customer_ID = Text.Combine({"C", Number.ToText(_, "00")}),
            /* Picks a name from the predefined list */
            Customer_Name = CustomerNames{_ - 1}, 
            /* Cycles through regions */
            Customer_Region = Regions{Number.Mod(_, List.Count(Regions))} 
        ])
    ),
    DataTypeChange = Table.TransformColumnTypes(Source, { {"Customer_ID", type text}, {"Customer_Name", type text}, {"Customer_Region", type text} })
in
DataTypeChange

Result:

3) dim_Products: 
The
products dimension with sample list of Products

let
    Products = {
        [Product_ID="P01", Prod_Name="LED TV", Prod_Category="Electronics", Prod_Price=500, Prod_Cost=300],
        [Product_ID="P02", Prod_Name="Smartphone", Prod_Category="Electronics", Prod_Price=800, Prod_Cost=500],
        [Product_ID="P03", Prod_Name="Laptop", Prod_Category="Electronics", Prod_Price=1200, Prod_Cost=900],
        [Product_ID="P04", Prod_Name="Refrigerator", Prod_Category="Home Appliances", Prod_Price=1500, Prod_Cost=1000],
        [Product_ID="P05", Prod_Name="Washing Machine", Prod_Category="Home Appliances", Prod_Price=1800, Prod_Cost=1400],
        [Product_ID="P06", Prod_Name="Microwave Oven", Prod_Category="Home Appliances", Prod_Price=600, Prod_Cost=400],
        [Product_ID="P07", Prod_Name="Sofa", Prod_Category="Furniture", Prod_Price=2000, Prod_Cost=1500],
        [Product_ID="P08", Prod_Name="Wooden Table", Prod_Category="Furniture", Prod_Price=900, Prod_Cost=600],
        [Product_ID="P09", Prod_Name="Wooden Chair", Prod_Category="Furniture", Prod_Price=2500, Prod_Cost=1900]
    },
    Source = Table.FromList(Products, Record.FieldValues, {"Product_ID", "Product_Name", "Category", "Price", "Cost"})
in
    Source

Result:

4) fact_Sales_Orders: The fact table with random sample of Order details data.

let
    SalesRecords = List.Transform({1..30}, each 
    let
        /* Generate a Range of Random Dates */
        DateInfo
            let
                StartDate = #date(2021, 1, 1),
                EndDate = #date(2021, 3, 31),
                RandomDaysOffset = Number.RoundDown(
                    Number.RandomBetween(0, Duration.Days(EndDate - StartDate))),
                RandomDate = Date.AddDays(StartDate, RandomDaysOffset),
                Date_ID = Number.FromText(Date.ToText(RandomDate, "yyyyMMdd")),
                Month = Date.Month(RandomDate)
            in
                [Order_Date = RandomDate, Order_Month = Month],
        
        /* Random Product selection */
        RandomProduct = Number.RandomBetween(1, 9)
    in
        [
        Order_ID = _,
        Product_ID = 
        /* 
        Exclude specific products for certain months:
        - For January (Order_Month = 1), exclude products P01, P02, and P03.
        - For February (Order_Month = 2), exclude products P04, P05, and P06.
        - For March (Order_Month = 3), exclude products P07, P08, and P09.
        If the RandomProduct falls within the exclusion list for the respective month,
        then the Product_ID is set to null (excluded from the data). 
        Otherwise, it assigns the product.
        */
        if                 
            (DateInfo[Order_Month] = 1 and List.Contains({1, 2, 3}, RandomProduct)) or
            (DateInfo[Order_Month] = 2 and List.Contains({4, 5, 6}, RandomProduct)) or
            (DateInfo[Order_Month] = 3 and List.Contains({7, 8, 9}, RandomProduct))
        then null
        else Text.Combine({"P", Number.ToText(RandomProduct, "00")}),
        Customer_ID = Text.Combine({"C", Number.ToText(Number.RandomBetween(1, 10), "00")}),
        
        Order_Date = DateInfo[Order_Date],
        Quantity_Sold = Number.RandomBetween(1, 5),
        Revenue = Number.RandomBetween(500, 3000),
        Profit = Number.RandomBetween(50, 500)
        ]
    ),
    /* Remove records where Product_ID is null */
    FilteredRecords = Table.SelectRows(Table.FromRecords(SalesRecords), each [Product_ID] <> null),
    /* Adjust column data types */
    DataTypeChange = Table.TransformColumnTypes(FilteredRecords,{{"Order_ID", Int64.Type}, {"Product_ID", type text}, {"Customer_ID", type text}, {"Order_Date", type date}, {"Quantity_Sold", Int64.Type}, {"Revenue", type number}, {"Profit", type number}})
in
    DataTypeChange

Result:
Please note that, the random values will keep change on each refresh of the above Queries.


--------------------------------------------------------------------------------------------------------
Thanks, TAMATAM ; Business Intelligence & Analytics Professional
--------------------------------------------------------------------------------------------------------

Tuesday, March 4, 2025

How to return the Metadata of Table or Dataset using Power Query

How to Check the Cardinality of Columns of a Table or Dataset using Power Query
Scenario:
Suppose we have a Table or Dataset "fct_Orders_Sample" in the Data Model. From this Table, we want to return the following Metadata:
"No. of Columns", "Column Name", "Column Type", "Column Size", "Column Cardinality" 

We can achieve this Scenario, using the following Power M-Query:

let
    /* Define your table name here */
    TableName = "fct_Orders_Sample",  

    /* Use the actual table data here */
    Source = #"fct_Orders_Sample",

    /* Retrieve the schema of the table */
    Schema = Table.Schema(Source),
    ColumnNames = Schema[Name],
    ColumnTypes = Schema[Kind],
    
    /* Efficiently calculate the column sizes */
    ColumnSizes = List.Transform(ColumnNames, each 
        let
            ColumnData = Table.Column(Source, _),
            DataSize = List.Sum(List.Transform(ColumnData, 
                each if _ is null then 0 else Binary.Length(Text.ToBinary(Text.From(_)))))
        in
            Number.Round(DataSize / (1024 * 1024), 4)  /* Convert size to MB */
        ),

    /* Calculate the total size of the table */
    TotalSize = Number.Round(List.Sum(ColumnSizes), 4),

    /* Calculate the cardinality for each column */
    Cardinality = List.Transform(ColumnNames, each List.Count(List.Distinct(Table.Column(Source, _)))),

    /* Find the maximum cardinality and its corresponding column */
    MaxCardinality = List.Max(Cardinality),
    MaxCardinalityColumn = ColumnNames{List.PositionOf(Cardinality, MaxCardinality)},

    /* Retrieve the last refresh date from the table properties */
    RefreshInfo = Value.Metadata(Source),
    LastRefreshDate = Record.FieldOrDefault(RefreshInfo, "RefreshDate", DateTime.LocalNow()),

    /* Create a metadata table from the lists */
    MetadataTable = Table.FromColumns(
        {List.Repeat({TableName}, List.Count(ColumnNames)), ColumnNames, 
        ColumnTypes, List.Repeat({List.Count(ColumnNames)}, List.Count(ColumnNames)), 
        ColumnSizes, List.Repeat({TotalSize}, List.Count(ColumnNames)), 
        Cardinality, List.Repeat({MaxCardinality}, List.Count(ColumnNames)), 
        List.Repeat({MaxCardinalityColumn}, List.Count(ColumnNames)), 
        List.Repeat({LastRefreshDate}, List.Count(ColumnNames))}, 
        {"TableName", "ColumnName", "ColumnType", "NoOfColumns", 
        "ColumnSizeInMB", "TotalSizeOfTableInMB", "ColumnCardinality", 
        "MaxCardinality", "MaxCardinalityColumn", "LastRefreshDate"}
    )
in
    MetadataTable

Result:


--------------------------------------------------------------------------------------------------------
Thanks, TAMATAM ; Business Intelligence & Analytics Professional
--------------------------------------------------------------------------------------------------------

Sunday, March 2, 2025

How to load data from a GraphQL endpoint into Power BI using Power Query

How to use Power Query to load data from a GraphQL endpoint into Power BI
GraphQL is a query language for APIs, and it provides a more efficient, powerful, and flexible alternative to REST APIs. 
It allows clients to request only the data they need and combine data from multiple sources into a single query. 
With GraphQL, you define the structure of the data you need, and the server returns exactly that, making it a perfect fit for modern applications.

Loading Data from GraphQL to Power BI:
Below is the M code (Power Query language) example that loads data from a GraphQL endpoint into Power BI. 
This example uses the GraphQL endpoint https://countries.trevorblades.com/ to get a list of countries, with their codes, and capitals.

let
    /* Define the source from the GraphQL endpoint */
    Source = Web.Contents (
        /* "https://your-graphql-endpoint.com/graphql", */
        "https://countries.trevorblades.com/",
        [
            Headers = [
                #"Method" = "POST",
                #"Content-Type" = "application/json"
             /* #"Authorization" = "Bearer your_access_token" */
                    ],
            Content = Text.ToBinary("{""query"": ""{ countries { code name capital } }""}")
        ]
    ),
    
    /* Parse the JSON response */
    JSON_Source = Json.Document(Source),
    SrcData = JSON_Source[data],
    CountriesList = SrcData[countries],
    
    /* Convert the list of countries to a table */
    Convert2Table = Table.FromList(CountriesList, Splitter.SplitByNothing(), null, null, ExtraValues.Error),
    
    /* Expand the records to get the country details */
    ExpandTable
        Table.ExpandRecordColumn(Convert2Table, 
            "Column1", {"code", "name", "capital"}, {"CountryCode", "CountryName", "CapitalCity"}
        ),
    
    /* Filter rows to include only specific countries */
    FilterRows = Table.SelectRows(ExpandTable, each ([CountryName] = "Australia" or [CountryName] = "Austria" or 
        [CountryName] = "Belgium" or [CountryName] = "Brazil" or [CountryName] = "France" or [CountryName] = "India" or 
        [CountryName] = "Italy" or [CountryName] = "Japan" or [CountryName] = "Mexico" or [CountryName] = "Singapore" or 
        [CountryName] = "Sweden" or [CountryName] = "Switzerland" or [CountryName] = "United Kingdom" or [CountryName] = "United States"))
    
in
    FilterRows

Result:

-----------------------------------------
Notes:
GraphQL Endpoint: In the above example, we have used below public GraphQL API is to fetch a list of countries along with their capital cities.
https://countries.trevorblades.com/

GraphQL Query: The following query fetches a list of countries along with their country codes and capital cities.
{ countries { code name capital } }

Headers: The headers include the HTTP method (POST) and content type (application/json). 
Authorization: 
This is the header key used to send authentication credentials.
"Authorization" = "Bearer your_access_token"
Bearer: 
This indicates the type of token being used; in this case, a bearer token. The Bearer tokens are commonly used in OAuth 2.0, a protocol for authorization, where the token is issued by an authentication server and then presented to the resource server to gain access to protected resources.
your_access_token: 
This is the actual token that grants access to the protected resource. If your API requires an authorization token, replace your_access_token with your actual token. For public APIs the authorization header can be omitted.

Content: The content includes the GraphQL query in JSON format.

#--------------------------------------------------------------Thanks--------------------------------------------------------------#

Tuesday, February 18, 2025

How to read data from a JSON File into Power BI Data Model

How to Load data from a JSON File into Power BI Data Model
Scenario:
Suppose we have a JSON File with sample data as per below, which we want to load into the Power BI Data Model.


JSON Data:
--------------------------------------------------------------------------------------------------------

{

  "data": [

    {

      "Order_ID": "1001",

      "Order_Date": "2025-01-01",

      "Product_Segment": "Electronics",

      "Product_Name": "Laptop",

      "Country": "USA",

      "Sales": 1500.00

    },

    {

      "Order_ID": "1002",

      "Order_Date": "2025-01-02",

      "Product_Segment": "Furniture",

      "Product_Name": "Office Chair",

      "Country": "Canada",

      "Sales": 300.00

    },

    {

      "Order_ID": "1003",

      "Order_Date": "2025-01-03",

      "Product_Segment": "Electronics",

      "Product_Name": "Smartphone",

      "Country": "Germany",

      "Sales": 800.00

    },

    {

      "Order_ID": "1004",

      "Order_Date": "2025-01-04",

      "Product_Segment": "Appliances",

      "Product_Name": "Refrigerator",

      "Country": "India",

      "Sales": 1200.00

    },

    {

      "Order_ID": "1005",

      "Order_Date": "2025-01-05",

      "Product_Segment": "Apparel",

      "Product_Name": "T-shirt",

      "Country": "Australia",

      "Sales": 50.00

    },

    {

      "Order_ID": "1006",

      "Order_Date": "2025-01-06",

      "Product_Segment": "Electronics",

      "Product_Name": "Tablet",

      "Country": "UK",

      "Sales": 600.00

    },

    {

      "Order_ID": "1007",

      "Order_Date": "2025-01-07",

      "Product_Segment": "Furniture",

      "Product_Name": "Desk",

      "Country": "France",

      "Sales": 400.00

    },

    {

      "Order_ID": "1008",

      "Order_Date": "2025-01-08",

      "Product_Segment": "Appliances",

      "Product_Name": "Washing Machine",

      "Country": "Japan",

      "Sales": 900.00

    },

    {

      "Order_ID": "1009",

      "Order_Date": "2025-01-09",

      "Product_Segment": "Apparel",

      "Product_Name": "Jacket",

      "Country": "Brazil",

      "Sales": 120.00

    },

    {

      "Order_ID": "1010",

      "Order_Date": "2025-01-10",

      "Product_Segment": "Electronics",

      "Product_Name": "Smartwatch",

      "Country": "Taiwan",

      "Sales": 200.00

    }

  ]

}

--------------------------------------------------------------------------------------------------------

We can use the default Power BI JSON Connector to load the data from the JSON File.
The Power Query steps in involved in loading the JSON dataset is as follows:

let

    Source = Json.Document(File.Contents("C:\Users\Tamatam\Desktop\MyLearn\Power BI\Inputs\ds_Sample_Orders.json")),

    Convert2Table = Table.FromRecords({Source}),

    ExpandListColumns = Table.ExpandListColumn(Convert2Table, "data"),

    ExpandColumnRecords = Table.ExpandRecordColumn(ExpandListColumns, "data", {"Order_ID", "Order_Date", "Product_Segment", "Product_Name", "Country", "Sales"}, {"Order_ID", "Order_Date", "Product_Segment", "Product_Name", "Country", "Sales"}),

    ChangeType = Table.TransformColumnTypes(ExpandColumnRecords,{{"Order_ID", Int64.Type}, {"Order_Date", type date}, {"Product_Segment", type text}, {"Product_Name", type text}, {"Country", type text}, {"Sales", Int64.Type}})

in

    ChangeType

Result:


--------------------------------------------------------------------------------------------------------
Thanks, TAMATAM ; Business Intelligence & Analytics Professional
--------------------------------------------------------------------------------------------------------

Friday, February 7, 2025

How to use Encode and Decode methods for binary-to-text conversions in Power Query

How to Encode and Decode the binary-to-text conversions using M-Query in Power BI
In Power Query, Base64 Encoding is a standard method for encoding binary data into Base64 encoded text format using ASCII characters.

1) Encoding Text:
Base64 Encoding is a way to represent binary data (like images or files) in a text format using only ASCII characters. It's called Base64 because it uses 64 different characters to represent the data.

Please Note: 
This method should not be used as a secure method for protecting sensitive information. It is simply a way to represent binary data in an ASCII string format. Anyone who can decode Base64 can easily get the original data.

How Base64 Encoding Works:
a) Convert to Binary: First, the text is converted to binary data.
b) Split into Groups: The binary data is split into groups of 6 bits.
c) Map to Characters: Each group is mapped to one of the 64 characters in the Base64 alphabet (A-Z, a-z, 0-9, +, and /).

Example:
let
    text2Encode = "Sample_1#X2$y#3&Z_9@a8#B&7C_Key",
    binaryData = Text.ToBinary(text2Encode),
    base64Encoded = Binary.ToText(binaryData, BinaryEncoding.Base64)
in
    base64Encoded

Result:
U2FtcGxlXzEjWDIkeSMzJlpfOUBhOCNCJjdDX0tleQ==

2) Decoding Text:
Base64 Decoding is the reverse process of Base64 encoding. It converts the Base64 encoded text back into its original binary form(like an image, file, or text).

How Base64 Decoding Works:
a) Map to Binary: Each character in the Base64 encoded text is mapped back to its 6-bit binary representation.
b) Combine Groups: The binary groups are combined into the original binary data.
c) Convert to Text: Finally, the binary data is converted back to the original text or file.

Example:
let
    base64Text = "U2FtcGxlXzEjWDIkeSMzJlpfOUBhOCNCJjdDX0tleQ==",
    binaryData = Binary.FromText(base64Text, BinaryEncoding.Base64),
    decodedText = Text.FromBinary(binaryData)
in
    decodedText

Result:
Sample_1#X2$y#3&Z_9@a8#B&7C_Key

#--------------------------------------------------------------Thanks--------------------------------------------------------------#

Thursday, February 6, 2025

How to Pivot and combine List of Items per Segment using M-Query in Power BI

How to use Text.Combine to combine List of Items per Segment using Power Query
Scenario:
Suppose we have a sample table with the columns as Product_Name, Product_Segment and Country as per below:


Now based on the above table, we want to Piovt the data by Country and show the combined list of Products per Product Segment separated by a delimiter (|) for each Country.
We can achieve this Scenario using the Text.Combine function as shown below:

let
    Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("jZDBDoIwEER/hfTMT6gR0YSLIfFAOKztBjbWhWzB+PlaY1ICAp6adt50ZrcoVI4WH+SoYRWrvUXdScOknb9xZYGNKuNCpSCG3G2ROTd9hzJBEgHWuPjLH8iRDcFSTAB2NZDXk16Yul5w1HOLZqQO8nO4WpyXp96Q+8sa1KnzBC3wrDOoiZCp/FPa3DHatK0l32e8/gu4mriKMtDvc53P6IkSHYTYfNY5pQdzr8PfvuUL", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [Prod_Name = _t, Prod_Segment = _t, Country = _t]),

    ChangeType = Table.TransformColumnTypes(Source,{{"Prod_Name", type text}, {"Prod_Segment", type text}, {"Country", type text}}),

    PivotByCountry = Table.Pivot(ChangeType, List.Distinct(ChangeType[Country]), "Country", "Prod_Name", each Text.Combine(_,"|"))
in
    PivotByCountry

Result:

--------------------------------------------------------------------------------------------------------
Thanks, TAMATAM ; Business Intelligence & Analytics Professional
--------------------------------------------------------------------------------------------------------

Wednesday, December 4, 2024

How to calculate Running Totals using Power Query in Power BI

How to calculate Cumulative Totals using M-Query in Power BI
Scenario:
Suppose we have a Sample Dataset as follows:


Based on above Data, we need to calculate the Running Totals for Qnty_Sold by Prod_Segment
We can achieve this Scenario using the following M-Query:

let
    Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("hZJPawIxEMW/iuxZMH8mk+RYS60teClCD+Ih3Q0aus1KVks/frPLZkGD6+klkN+8zLzZ7Qq6YAtGGCnmhWAMIOrW1vbXta7x8fJS2/IcGu/Ktrv5Q218FU8oi/28w1XCOQeuoq5NqFz7PcEKOrCjNddKYNSP5nK2IUNXwfjSxgPgQOpEMkb0lOuIIgwoHVGKMIW++cqZqJoPJE8kgqQT300gkAGEBGpJuIz6fDSu41aX4N35EuzVfBTLOAqUR13a6oYa+6MJwjFPQqAz25qv2t7DeMpRJExx7LPIvVJfPFnJxAAQKe5YJUpmqTPEPrvc6d2cTLd9iNkk4pbhHadEgchGEY1U37arDt3TdfNjZ0+nU+26OVyvJ81WWwymn6Y9On+YbUwZ9WEdvK2jQWsWdeP+bJi9Buerfn3yKmM+Qt32IonqY31cJM1DxYb2/w==", BinaryEncoding.Base64), Compression.Deflate)), 
let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [Order_Date = _t, Order_Id = _t, Prod_Name = _t, Prod_Segment = _t, Country = _t, Qnty_Sold = _t]
    ),

ChangeType =  Table.TransformColumnTypes(Source,{{"Order_Date", type date}, {"Order_Id", Int64.Type}, {"Prod_Name", type text}, {"Prod_Segment", type text}, {"Country", type text}, {"Qnty_Sold", Int64.Type}}),
    
Add_Index = Table.AddIndexColumn(ChangeType, "Index_No", 1, 1, Int64.Type),

/* Running Total Sales*/
Run_Sales = Table.AddColumn(Add_Index, "RT_Sales", 
        each List.Sum(
                List.FirstN(  
                    Add_Index[Qnty_Sold],
                    [Index_No]
                    )
                )   
            ),

/* Running Total Sales by Product Segment */
Seg_Run_Sales = Table.AddColumn(Run_Sales, "Segment_RT_Sales", 
        (OutTable) => 
        List.Sum(
            Table.SelectRows(Add_Index, 
                (InTable) => InTable[Index_No] <= OutTable[Index_No]
                and 
                InTable[Prod_Segment] = OutTable[Prod_Segment]
                )
            [Qnty_Sold]
            )
        )
in
    Seg_Run_Sales

Result:

--------------------------------------------------------------------------------------------------------
Thanks, TAMATAM ; Business Intelligence & Analytics Professional
--------------------------------------------------------------------------------------------------------

Sunday, May 19, 2024

How to Rank the Data for each Group in Power Query

How to apply Rank for each Group of Data in Power Query
Scenario:
Suppose we have the Sample dataset with columns as Product Segment, Product Name and Order Quantity as follows:


Now we want to apply the Ranking for the Order Quantity sort by Descending for each Product Segment sort by Ascending Order.
We can achieve this Scenario using the following Power Query:

let
    SourceData = "dY49C8IwEIb/SsncQaioq1alCF2k4FAynOlBD2Ou5EP8+cYMVaxd7ut5795rW3HQqLxlQ8qJXFRgu2xPTsW6EDL/5WfoiGNe/mMcPNoZ2KDGBzliE5t1EhyDNeSDxTjZYRfjajIve6DPyW/SwFXjuFPxHbPtMGgCo/BteAHX16B6Mjg6TlUl8y39vJkR1PRMvJjhp0AqCRZCyhc=",
    Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText(SourceData, BinaryEncoding.Base64), Compression.Deflate)), 
        let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [ProdSegment = _t, ProdName = _t, OrderQnty = _t]),
    ChangedType = Table.TransformColumnTypes(Source,{{"ProdSegment", type text}, {"ProdName", type text}, {"OrderQnty", Int64.Type}}),
    SegmentGroupData = Table.Group(ChangedType, {"ProdSegment"}, {{"ProdSegGroup", 
        each _, type table [ProdSegment=nullable text, ProdName=nullable text, OrderQnty=nullable number]}}),
    RankQnty = Table.AddColumn(SegmentGroupData, "RankByProdQnty", each Table.AddRankColumn([ProdSegGroup], 
        "RankProdQnty",{{"ProdSegment",Order.Ascending},{"OrderQnty",Order.Descending}},[RankKind = RankKind.Dense])),
    SelectColumns = Table.SelectColumns(RankQnty,{"RankByProdQnty"}),
    ExpandRankByProdQnty = Table.ExpandTableColumn(SelectColumns, "RankByProdQnty", 
        {"ProdSegment", "ProdName", "OrderQnty", "RankProdQnty"}, 
        {"Prod_Segment", "Prod_Name", "Order_Qnty", "Rank_ProdQnty"})
in
    ExpandRankByProdQnty 

Result:


Notes:
To apply Rank with Dense: [RankKind = RankKind.Dense]
To apply Rank without Dense: [RankKind = RankKind.Competition]
To apply Rank based on source data Order: [RankKind = RankKind.Ordinal]


--------------------------------------------------------------------------------------------------------
Thanks, TAMATAM ; Business Intelligence & Analytics Professional
--------------------------------------------------------------------------------------------------------

Featured Post from this Blog

How to compare Current Snapshot Data with Previous Snapshot in Power BI

How to Dynamically compare two Snapshots Data in Power BI Scenario: Suppose we have a sample Sales data, which is stored with Monthly Snapsh...

Popular Posts from this Blog