How to generate additional columns based on existing columns using M-Query in Power BI

How to generate additional % columns based on existing columns using List. Accumulate () Function in Power Query

Scenario:
Suppose we have a sample data with below fields:
[Year], [Month], [Product], [Gross_Sales], [COGS], [Operating_Expenses], [Interset_Taxes]

Now, based on the [Gross_Sales] column, we need to generate the expense % columns as per below: 

[% COGS], [% Operating_Expenses], [% Interset_Taxes]

We can achieve this Scenario using the following M-Query:

let

    /* Original data source: */

    Source = Table.FromRows(

        Json.Document(

            Binary.Decompress(

                Binary.FromText("hZGxDsIwDET/pXOHxImd5AcYECyIreoQRIdKiCJAfD/xDVVpVbpcpOtd+uw0TUWGTFVX+3wvesrXfihnir6oS1Dv4VRtPaZ33aXoubt1n/7VD1pl1hgRFbUoMk8rx/wsesiP9/DQj05NsohHgTON//KIZ01zxC80LbzBI6xXk1hU4qIy4wFNSqhoMfA6TcClzmBgETgbNDEQVplQZDh/aISA7TGvDXDWedhgVKMdG7BLs8ETsEpnHEbgxQhzHiCTp/HNygra9gs=", BinaryEncoding.Base64),

                Compression.Deflate

            )

        ),

        let _t = ((type nullable text) meta [Serialized.Text = true]) in

        type table [

            Year = _t, Month = _t, Product = _t, Gross_Sales = _t,

            COGS = _t, Operating_Expenses = _t, Interest_Taxes = _t

        ]

    ),

    /* Change column data types: */

    ChangeType = Table.TransformColumnTypes(Source, {

        {"Year", Int64.Type}, {"Month", type text}, {"Product", type text},

        {"Gross_Sales", Int64.Type}, {"COGS", Int64.Type},

        {"Operating_Expenses", Int64.Type}, {"Interest_Taxes", Int64.Type}

    }),

    /* List of columns based on which % columns will be generated: */

    Cols_List = {"COGS", "Operating_Expenses", "Interest_Taxes"},

    /* Generate additional Percentage Columns */

    Add_Ratios = List.Accumulate(

        Cols_List,

        ChangeType,

        (i, c) => Table.AddColumn(

            i,

            "% " & c,

            each if ([Gross_Sales] = null or [Gross_Sales] = 0) then null

                 else (Record.Field( _, c) / [Gross_Sales]),

            type nullable Percentage.Type

        )

    )

in

    Add_Ratios

--------------------------------------------------------------
Notes:

In the above List.Accumulate(i,c) function, the variables will work as per below:
c: The current column name from Cols_List being processed in that iteration.
i: The intermediate table result that accumulates with each added column. It starts as ChangeType and grows with each new % Column.

If Cols_List = {"COGS", "Operating_Expenses"}, then the loop will work as:

First iteration: c = "COGS"

Adds column % COGS to ChangeType, result stored in i

Second iteration: c = "Operating_Expenses"

Adds column % Operating_Expenses to the table from step 1

And so on, until all columns in Cols_List have their corresponding % columns added.

Output:

--------------------------------------------------------------

The above M-Query Code can also be return as per below:

let

    /* Original data source: */

    Source = Table.FromRows(

        Json.Document(

            Binary.Decompress(

                Binary.FromText("hZGxDsIwDET/pXOHxImd5AcYECyIreoQRIdKiCJAfD/xDVVpVbpcpOtd+uw0TUWGTFVX+3wvesrXfihnir6oS1Dv4VRtPaZ33aXoubt1n/7VD1pl1hgRFbUoMk8rx/wsesiP9/DQj05NsohHgTON//KIZ01zxC80LbzBI6xXk1hU4qIy4wFNSqhoMfA6TcClzmBgETgbNDEQVplQZDh/aISA7TGvDXDWedhgVKMdG7BLs8ETsEpnHEbgxQhzHiCTp/HNygra9gs=", BinaryEncoding.Base64),

                Compression.Deflate

            )

        ),

        let _t = ((type nullable text) meta [Serialized.Text = true]) in

        type table [

            Year = _t, Month = _t, Product = _t, Gross_Sales = _t,

            COGS = _t, Operating_Expenses = _t, Interest_Taxes = _t

        ]

    ),

    /* Change column data types: */

    ChangeType = Table.TransformColumnTypes(Source, {

        {"Year", Int64.Type}, {"Month", type text}, {"Product", type text},

        {"Gross_Sales", Int64.Type}, {"COGS", Int64.Type},

        {"Operating_Expenses", Int64.Type}, {"Interest_Taxes", Int64.Type}

    }),

    /* Define a Temp Table with flags to skip or keep columns: */

    tbl_ColumnFlags = #table(

        {"Field_Name", "Skip_Flag"},

        {

            {"Year", true},

            {"Month", true},

            {"Product", true},

            {"Gross_Sales", true},

            {"COGS", false},

            {"Operating_Expenses", false},

            {"Interest_Taxes", false}

        }

    ),

    /* Get list of columns where Skip = false: */

    Cols_List = Table.SelectRows(tbl_ColumnFlags, each [Skip_Flag] = false)[Field_Name],

    /* Generate additional Percentage Columns */

    Add_Ratios = List.Accumulate(

        Cols_List,

        ChangeType,

        (i, c) => Table.AddColumn(

            i,

            "% " & c,

            each if ([Gross_Sales] = null or [Gross_Sales] = 0) then null

                 else (Record.Field( _, c) / [Gross_Sales]),

            type nullable Percentage.Type

        )

    )

in

    Add_Ratios

--------------------------------------------------------------------------------------------------------

Thanks, TAMATAM ; Business Intelligence & Analytics Professional

--------------------------------------------------------------------------------------------------------

How to display values of a Selected Field in the Slicer using Field Parameters in Power BI

How to display values of a Default Field in the Slicer using Field Parameters in Power BI

Scenario:
When a Field is selected from Field Parameter Slicer, then we want to display the values of that Selected Field in the Slicer.

Example: 

When a [Product] dimension is selected then we want to display the values of that dimension {Radio, Fridge, Television etc}.

When no Dimension is selected then we want to display the values of the first Dimension in the Field Parameters.

Example: 

Say [Month-Year] is the first dimension in the Field Parameters list, then we want to display the values of that dimension {Jan-2024, Feb-2024, March-2024 etc}.

We can achieve this Scenario, as explained below.

1) Create a Field Parameters with a list of Dimensions:

First, we need to create a Field Parameters table with required Dimensions as per below.

fp_Dim_Selector = {

    ("Month-Year", NAMEOF('dim_Calendar'[Month_Year]), 1,"Month_Year"),

    ("Region", NAMEOF('dim_Region'[Region_Name]), 2,"Region_Name"),

    ("Country", NAMEOF('dim_Country'[Country_Name]), 3,"Country_Name"),

    ("Customer", NAMEOF('dim_Customer'[Customer_Name]), 4,"Customer_Name"),

    ("Product", NAMEOF('dim_Products'[Product_Name]), 5,"Product_Name")

}

2) Create the Slicers for Dimension and Its Values:

We can create the "Dimension" Slicer using Field_Name as Source.

Next we can create the "Dimension Details" Slicer using Field_Name and enabling the option to "Show values of selected field" by right clicking on Field_Name.

In general, the Details Slicer will show values of the Selected Dimension. However, when No Dimension is selected from Dimension Slicer, then the Details Slicer will fail to show the values, as there is no Dimension Selected, as shown below:

Instead of showing Error, by default we want to display the values of first Dimension Month-Year {Jan-2024, Feb-2024, March-2024 etc}.

This can be achieved as explained in the next step.

3) By default, display the Values of first Dimension:

To achieve this, we can create a below Measure to capture the Minimum Index Number of the Dimension.

Selected Min Dim Index =

VAR _Dim_Index = {1,2,3,4,5}

VAR _Min_Dim_Index =

CALCULATE(MIN(fp_Dim_Selector[Field_Index]),

    fp_Dim_Selector[Field_Index] IN _Dim_Index

    )

RETURN

_Min_Dim_Index

Next, we need to use this Measure as a Visual Level filter for Details Slicer, using Top N and Bottom 1 filter method as shown below:

Result:

--------------------------------------------------------------------------------------------------------

Thanks, TAMATAM ; Business Intelligence & Analytics Professional

--------------------------------------------------------------------------------------------------------

How to load data from a Data Warehouse in Azure Databrics Catalog using Power BI

How to load data from a Data Warehouse in Azure Databrics Catalog with Native SQL Query using Power BI

Unity Catalog:
It is a data governance solution in Databricks that provides centralized access control, auditing, lineage tracking, and data discovery across multiple workspaces. It ensures secure and organized data management.

Catalog:

It is a top-level container within Unity Catalog that holds multiple Databases (Schemas). It helps in data isolation and logical organization.
Example: A SalesData catalog may contain databases like Retail, Wholesale, E-commerce.

Databricks primarily operates on a Lakehouse architecture, which combines elements of data lakes and data warehouses. While Databricks SQL Warehouses are used for querying and analytics, Databricks also supports Databases (Schemas) within Unity Catalog for structured data organization.

SQL Warehouses: 

Compute resources optimized for running SQL queries on structured data. Supports BI tools, Delta Lake integration, and fast query execution.

Databases (Schemas): 

The Database (Schema) exists inside a Catalog and is used for logical grouping of related data. It contains tables, views, functions, and models.
Example: Inside the Retail database, you might store tables like Customers, Orders, and Products.

Notes:
When connecting to Databricks Unity Catalog, Power Query may flatten the hierarchy and display the Catalog as a Database.
This happens because traditional databases (like SQL Server) don’t have a Catalog layer, so Power Query maps the Catalog to a Database for compatibility.

Since Databricks Unity Catalog introduces a three-tier hierarchy (Catalog → Database → Table/View), Power Query may simplify this by treating the Catalog as a Database.

Databricks does not function like a traditional database system (such as SQL Server or PostgreSQL), but it does provide database-like structures within Unity Catalog for managing data efficiently.

--------------------------------------------------------------------------------------------------

Scenario:

To connect and load data from a Data Warehouse in Azure Databrics Catalog, we need the following details like, Server Hostname, HTTP Path, which are mandatory.

I have defined the Parameters in the Power Query to hold the Server Hostname, HTTP Path, Catalog/DB Name, and Schema name as shown below samples (Dummy values).

We will use these parameters in the following examples. Please make sure to pass your own correct values for these parameters while you are testing.

p_ServerHostName =  adb-server_host_id.3.azuredatabricks.net

p_http_Path = sql/protocolv1/o/server_host_id/cluster_id

p_Catalog_DB = MyCatlog_db

p_Schema = MySchema

p_Load_By_Year = 2021, This Parameter is used to limit the Data Load from the Query. 

Notes:

In the Azure Databrics Workspace, we can find the Server Hostname and HTTP path values from the following ways:

Compute >[Cluster_Name] > Advanced Options > JDBC/ODBC:

Server hostname: adb-server_host_id.3.azuredatabricks.net

HTTP path: sql/protocolv1/o/server_host_id/cluster_id

SQL Warehouses> [MyWarehouse_Name] > Connection Details:

Server hostname: adb-server_host_id.3.azuredatabricks.net 

HTTP path: /sql/1.0/warehouses/warehouse_id

1) Loading a View or Table from the Data Warehouse in Azure Databrics Catalog:

In Power BI, we use default "Azure Datarbics" connector to connect and load a View or Table from a Data Warehouse in Azure Databrics Catalog.

We need to pass only the Server Hostname and HTTP Path as shown below.

After Connecting to the Source, it will open a dialogue box with available Catalogs and their underlying Warehouses/Databases and Objects to select.

The following is the sample Power Query logic of an object (View) loaded, which is updated with Parameters defined above:

let

    Source = Databricks.Catalogs(p_ServerHostName,p_http_Path, [Catalog=null, Database=null, EnableAutomaticProxyDiscovery=null]),

    Src_Database = Source{[Name=p_Catalog_DB,Kind="Database"]}[Data],

    Src_Schema = Src_Database{[Name=p_Schema,Kind="Schema"]}[Data],

    Src_Object = Src_Schema{[Name="MyViewName",Kind="View"]}[Data]

in

    Src_Object

Note:
If you want to load Table instead of View, change the Kind="View" to Kind="Table"

2) Loading data using a Native SQL Query from the Data Warehouse in Azure Databrics Catalog:

To connect and load data by running a Native SQL Query from the Data Warehouse in Azure Databrics Catalog, we need to pass the Server Hostname, HTTP Path and Catalog Name are mandatory.

Please make sure include the Schema.Table/View Name in the Native SQL Query section.

The following is the sample Power Query logic of data loaded using Native SQL Query, which is updated with Parameters defined above:

let

    vCY = Date.Year(DateTime.LocalNow()),

    vYearsList = " IN (" & Number.ToText(vCY) & "," & Number.ToText(vCY-1) & "," & Number.ToText(vCY-2) & ")", 

    vLoadYears = if p_Load_By_Year is null then vYearsList else ">= " & Number.ToText(p_Load_By_Year),

    vSrc_Qry = "SELECT transact_id, transact_date, field3, field4, field5, fieldN

    FROM " & p_Schema & ".fact_Transactions fact

    Where YEAR(fact.transact_date) " & vLoadYears,

    Source = Value.NativeQuery(Databricks.Catalogs(p_ServerHostName, p_http_Path, 

    [Catalog=p_Catalog_DB, Database=null, EnableAutomaticProxyDiscovery=null]){[Name=p_Catalog_DB,Kind="Database"]}[Data], 

    vSrc_Qry, null, [EnableFolding=true]),

    ChangType = Table.TransformColumnTypes(Source,{{"transact_date", type date}})

in

    ChangType

Notes:
p_Load_By_Year is a Year Parameter is used to limit the Data Load from the Query. If this Parameter is Blank (), then the Data will be loaded by default for last 3 Years (from Current Year). Otherwise, it loads the Loads the data >= 2021 (Year value from Parameter)

--------------------------------------------------------------------------------------------------------

Thanks, TAMATAM ; Business Intelligence & Analytics Professional

--------------------------------------------------------------------------------------------------------