Showing posts with label Power_Query. Show all posts
Showing posts with label Power_Query. Show all posts

Saturday, March 7, 2026

How to use Power Query to Handle Multi‑Row Headers Dynamically from Excel to Power BI

How to use Power Query to Handle Composite Headers Dynamically from Excel to Power BI
Scenario:
Suppose we have a sample of FTE data in an Excel sheet where the actual headers begin at row number 4. However, the first three rows may contain additional text or values that provide context. For example, in a weekly allocation report, rows 1–3 might contain date ranges or descriptive labels above the actual header row, as shown below:


Considerations in Output:

When building the final table, we need to treat the first five columns differently from the last five:
  • Columns 1 to 5 (A to E): These are structural fields such as UserID, UserName, Project_ID, FTE_Alloc_Status, and FTE_Percent. For these, we must ignore any text or values in rows 1 to 3 and simply use the header from row 4.
  • Columns 6 to 10 (F to J): These represent weekly allocations (WK1 to WK5). For these, we must concatenate the text or values from rows 1 to 3 with the header in row 4 to form a composite header.
    • Example: WK1_30-12-2024_03-01-2025.
This approach ensures that the weekly columns carry both the identifier (WK1, WK2, etc.) and the contextual date ranges, making the dataset more meaningful and self-explanatory.

We can achieve this scenario, using the below Power Query:

let
    /* LoadWorkbookAndSheet: 
       Load the Excel workbook from the given source path and access the target sheet "FTE_Sample". 
    */
    Source = Excel.Workbook(File.Contents(SrcPath), null, true),
    SrcData_Sheet = Source{[Item="FTE_Sample",Kind="Sheet"]}[Data],

    /* FindHeaderRowIndex: 
       Dynamically detect the row index where actual headers begin. 
       This is identified by locating "UserID" in the first column.
    */
    HeaderRowIndex = List.PositionOf(SrcData_Sheet[Column1], "UserID"),

    /* ExtractHeaderAndPreHeaderRows: 
       Capture all rows up to the header row. 
       Separate the HeaderRow (row containing actual column names) 
       and PreHeaderRows (rows above the header that may contain contextual values). 
    */
    AllRowsToHeader = List.FirstN(Table.ToRows(SrcData_Sheet), HeaderRowIndex+1),
    HeaderRow = AllRowsToHeader{HeaderRowIndex},
    PreHeaderRows = List.RemoveLastN(AllRowsToHeader, 1),

    /* BuildCompositeHeaders: 
       Construct raw composite headers by combining HeaderRow with PreHeaderRows. 
       Non-null values are concatenated using an underscore (_) to form meaningful column names. 
    */
    RawCompositeHeaders = List.Transform(
        List.Zip({HeaderRow} & PreHeaderRows),
        each Text.Combine(
            List.Select(List.Transform(_, Text.From), (x) => x <> null and x <> ""),
            "_"
        )
    ),

    /* ApplySkipLogic: 
       Define columns that should not use composite headers. 
       For these columns (e.g., "UserID", "FTE_Percent"), retain only the HeaderRow value. 
    */
    SkipColumns = {"FTE_Percent", "UserID"},
    CompositeHeaders = List.Transform(
        List.Zip({HeaderRow, RawCompositeHeaders}),
        each if List.Contains(SkipColumns, _{0}) then _{0} else _{1}
    ),

    /* PromoteHeadersAndRename: 
       Promote the detected HeaderRow as column names. 
       Then replace them with CompositeHeaders to ensure contextual information is included 
       for selected columns while keeping structural columns clean. 
    */
    PromoteHeaders = Table.PromoteHeaders(
        Table.Skip(SrcData_Sheet, HeaderRowIndex), 
        [PromoteAllScalars=true]
    ),
    FinalTable = Table.RenameColumns(
        PromoteHeaders, 
        List.Zip({Table.ColumnNames(PromoteHeaders), CompositeHeaders})
    )
in
    FinalTable

Output:


--------------------------------------------------------------------------------------------------------
Thanks
--------------------------------------------------------------------------------------------------------

Tuesday, February 17, 2026

How to return the Last Refresh Date of SQL Tables in Power BI

How use Power Query to return the Last Refresh Date of SQL Tables in Power BI
Scenario:
We have a set of tables/views sourced from Azure SQL Synapse, each containing a column name as _modified_timestamp . Using this column, we need to generate a summary dataset in Power BI that returns the last refresh date for each table/view.

Src_Tables = {"dim_Country", "dim_Product", "dim_Segment"}

We can achieve this Scenario using the following Power Query:

let
    // Data Source Parameters:
    _server = "MyServer-synapse-server.database.windows.net",
    _database = "db_MyAnalytics",
    _schema = "dbo",

    // Source Connection:
    Source = Sql.Database(_server, _database),

    // List of tables for which we need to check last refresh Date:
    Src_Tables = {"dim_Country", "dim_Product", "dim_Segment"},
 
    // Function to run native query for each table:
    GetLastRefreshDate = (TableName as text) =>
        let
            Query = "SELECT MAX(_modified_timestamp) AS Last_Refresh_Date FROM " & _schema & "." & TableName,
            Query_Result = Value.NativeQuery(Source, Query),
            WithServer = Table.AddColumn(Query_Result, "Server_Name", each _server),
            WithDb = Table.AddColumn(WithServer, "Database_Name", each _database),
            WithSchema = Table.AddColumn(WithDb, "Schema_Name", each _schema),
            WithTable = Table.AddColumn(WithSchema, "Table_Name", each TableName),
            Result = Table.ReorderColumns(WithTable, {"Server_Name","Database_Name","Schema_Name","Table_Name","Last_Refresh_Date"})
        in
            Result,

    // Apply function to get the last refresh date for each table:
    All_Results = List.Transform(Src_Tables, each GetLastRefreshDate(_)),

    // Combine each table result into one table:
    Final_Result = Table.Combine(All_Results)
in
    Final_Result

Sample Result:


Notes: 
Use parameters to define source connection details such as Server, Database, and Schema. Avoid hard‑coding these values directly within the query definition.

We can also achieve the above Scenario using the following Power Query:

let
    // Connection Parameters:
    _server   = p_server,
    _database = p_database,
    _schema   = p_schema,

    // Data Source Connection
    _source = Sql.Database(_server, _database),

    // Tables to check for last Refresh Date:
    _src_tables = {
        "fact_Sales", 
        "fact_Inventory"
        },

    // Output Table schema definition:
    _schema_table = type table [
        Attribute_ID    = Int64.Type, 
        Entity_Name     = Text.Type,
        Entity_Type     = Text.Type,
        Attribute_Type  = Text.Type, 
        Attribute_Value = Any.Type
        ],

    // Function to get last refresh date for each Table:
    _get_last_refresh_date = ( _id as number, _table as text ) =>
        let
            _query = "SELECT MAX(_modified_timestamp) AS Last_Refresh_Date FROM " & _schema & "." & _table,
            _query_result = Value.NativeQuery(_source, _query),
            _last_date = if Table.IsEmpty(_query_result) then null else _query_result{0}[Last_Refresh_Date]
        in
            #table(_schema_table, { {_id, _table, "Table", "Data Refresh Date", _last_date} }),

    // Combine the result of each Table:
    _combined_result = Table.Combine(
        List.Transform({0..List.Count(_src_tables)-1}, each _get_last_refresh_date(_ + 1, _src_tables{_}))
    ),

    // Append the Power BI Data Model Refresh Date row:
    _final_result = Table.InsertRows(
        _combined_result,
        Table.RowCount(_combined_result),
        {
            [
                Attribute_ID    = Table.RowCount(_combined_result)+1,
                Entity_Name     = "Power BI Data Model",
                Entity_Type     = "Data Model",
                Attribute_Type  = "Data Refresh Date",
                Attribute_Value = DateTime.LocalNow()
            ]
        }
    )
in
    _final_result

Sample Result:

Notes:
We can use the below code to return the table with only a record for the Power BI Data Model refresh Date Time in CET.
let
    _CET_DateTime = DateTimeZone.RemoveZone(
                    DateTimeZone.SwitchZone(DateTimeZone.UtcNow(), 2)
                    ),

    _Source = #table(
        type table [
            Attribute_ID    = Int64.Type, 
            Entity_Name     = Text.Type,
            Entity_Type     = Text.Type,
            Attribute_Type  = Text.Type, 
            Attribute_Value = DateTime.Type
        ],
        {
            {1, "Power BI Data Model", "Data Model", "Data Refresh Date", _CET_DateTime }
        }
    )
in
    _Source


By default, the DateTime.LocalNow() will return the datetime of the Local Time Zone (IST for India).

We can use the below Power Query function to convert the Time Zone to CET (UTC+2 Hours):
DateTimeZone.SwitchZone(DateTimeZone.UtcNow(), 2)

The DateTimeZone.RemoveZone function will ensure to remove TimeZone value from result and keep only the DateTime value.

In DAX, the TODAY() function returns the Today Date. The NOW() function returns the UTC Date Time stamp by default.

We can use the below DAX logic to convert the UTC Datetime to CET:

CET Time Now = NOW()+TIME(2,0,0)

IST Time Now = NOW()+TIME(5,30,0)

--------------------------------------------------------------------------------------------------------
Thanks
--------------------------------------------------------------------------------------------------------

Thursday, February 12, 2026

How to Identify the duplicate rows in a table using Power Query

How to Identify the duplicate Values in a Column using Power Query
Scenario:
Suppose, we have a dataset(ds_Sample) as below with duplicate values in a Prod_ID column and some duplicate rows:
let
    Source = #table (
        {"Trans_ID", "Prod_ID", "Prod_Name"},
        {
            {12345, 123, "Laptop"},
            {12346, 124, "Radio"},
            {12347, 125, "Keyboard"},
            {12348, 126, "Television"},
            {12349, 127, "Printer"},
            {12350, 128, "Scanner"},
            {12351, 129, "Camera"},
            {12352, 130, "Tripod"},
            {12345, 123, "Laptop"},     // duplicate row
            {12348, 126, "Television"}, // duplicate row
            {12351, 129, "Camera"},     // duplicate row
            {12356, 127, "Printer"},
            {12357, 124, "Radio"},
            {12358, 126, "Television"}
        }
    ),
    ChangeType = Table.TransformColumnTypes(Source, {
        {"Trans_ID", Int64.Type},
        {"Prod_ID", Int64.Type},
        {"Prod_Name", type text}
    })
in
    ChangeType

Now can identify and flag the duplicate values and rows using below methods.

Method-1: Using Text.Combine method with Composite Key based on specific Columns:

let
    /* Source dataset */
    Source = ds_Sample,

    /* Columns to include in Composite Key */
    SelectedCols = {"Trans_ID","Prod_ID"},

    /* Composite Key from selected columns */
    Def_CompositeKey = Table.AddColumn(Source, "Composite_Key", 
        each Text.Combine(
            List.Transform(
                Record.ToList(Record.SelectFields(_, SelectedCols)), 
                each Text.From(_)
                ), 
                "|"
            )
        ),

    /* Flag duplicate rows using Composite Key */
    Flag_DupRow = Table.AddColumn(Def_CompositeKey, "IsDuplicate_Row", 
        each if List.Count(List.Select(Def_CompositeKey[Composite_Key], (x) => x = [Composite_Key])) > 1 
        then "Yes" else "No"
        ),

    /* Flag duplicate products by Prod_ID */
    Flag_DupProd = Table.AddColumn(Flag_DupRow, "IsDuplicate_Prod", 
        each if List.Count(List.Select(Source[Prod_ID], (x) => x = [Prod_ID])) > 1 
        then "Yes" else "No"
        )
in
    Flag_DupProd

Result:
------------------------------------------------
Method-2: Using Text.ToBinary method and Composite Key (Binary Encoded) based on specific Columns:

let
    // Source dataset
    Source = ds_Sample,

    // Columns to include in Composite_Hash_Key
    SelectedCols = {"Trans_ID","Prod_ID"},

    // Generate Composite_Hash_Key from selected columns only
    Def_CompositeKey = Table.AddColumn(Source, "Composite_Hash_Key", 
        each Binary.ToText(
            Text.ToBinary(
                Text.Combine(
                    List.Transform(
                        Record.ToList(Record.SelectFields(_, SelectedCols)), 
                        each Text.From(_)
                        ), 
                        "|"
                    )
                ),
                BinaryEncoding.Base64
            )
        ),

    // Flag duplicate rows using Composite_Hash_Key
    Flag_DupRow = Table.AddColumn(Def_CompositeKey, "IsDuplicate_Row", 
        each if List.Count(List.Select(Def_CompositeKey[Composite_Hash_Key], (x) => x = [Composite_Hash_Key])) > 1 
        then "Yes" else "No"
        ),

    // Flag duplicate products by ProdID
    Flag_DupProd = Table.AddColumn(Flag_DupRow, "IsDuplicate_Prod", 
        each if List.Count(List.Select(Source[Prod_ID], (x) => x = [Prod_ID])) > 1 
        then "Yes" else "No"
        )
in
    Flag_DupProd

Result:
------------------------------------------------
Method-3: Using Table.Group method to identify the Duplicate values based on Prod_ID:

let
    Source = ds_Sample,
    // Count how many times each ProdID appears
    GroupByProd = Table.Group(Source, {"Prod_ID"}, {{"Prod_Count", each Table.RowCount(_), Int64.Type}}),
    // Join back to original details table
    MergeToDetails = Table.NestedJoin(Source, {"Prod_ID"}, GroupByProd, {"Prod_ID"}, "ProdCount"),
    ExpandTable = Table.ExpandTableColumn(MergeToDetails, "ProdCount", {"Prod_Count"}),
    // Add a column to flag duplicate values
    FlagDup_Prod = Table.AddColumn(ExpandTable, "IsDuplicate_Prod", each if [Prod_Count] > 1 then "Yes" else "No")
in
    FlagDup_Prod

Result:
------------------------------------------------
Method-4: Using Table.Group method to return the no. of Duplicate rows and products:

let
    Source = ds_Sample,
    // 1. Count duplicate rows
    DistinctRows = Table.Distinct(Source),
    DuplicateRowCount = Table.RowCount(Source) - Table.RowCount(DistinctRows),

    // 2. Count duplicate products (ProdID)
    GroupByProd = Table.Group(Source, {"Prod_ID"}, {{"Count", each Table.RowCount(_), Int64.Type}}),
    DuplicateProd = Table.RowCount(Table.SelectRows(GroupByProd, each [Count] > 1)),

    // 3. Return summary of results
    Summary= #table(
        {"Metric", "Value"},
        {
            {"Duplicate Rows", DuplicateRowCount},
            {"Duplicate Products", DuplicateProd}
        }
    )
in
    Summary

Result:

--------------------------------------------------------------------------------------------------------
Thanks
--------------------------------------------------------------------------------------------------------

Saturday, November 22, 2025

How to Use Dynamic Query Parameters to Switch Source Query Connections in Power BI

How to Switch Source Query Connection Strings in Power BI with Dynamic Query Parameters
Scenario:
To manage multiple data or query environments such as Dev, Test, and Prod often requires maintaining separate connection strings in Power BI. 
Rather than hard‑coding server, database, and schema names, you can centralize these details in an environment table and control them dynamically through a parameter (e.g. p_Query_Env with values like Dev, Test, and Prod). 
With this setup, switching between environments becomes as simple as changing the parameter value, while Power Query automatically selects the correct server, database, and schema. This provides a clean, scalable solution for environment management that minimizes manual effort and ensures consistency across deployments.

We can achieve the above Scenario, as discussed below.

Step1: Define an Environment Variables Table
To enable dynamic source data environment switching in Power BI, we need to create a dedicated Environment Variables Table like below. This table acts as a central repository for all connection details across your environments (Dev, Test, and Prod).

Info_Query_Environment:

let
    /* Define Data Environment variables table with respective connection details */
    _Env_Table = 
        #table(
            {"Source_Env", "Server_Name", "DB_Name", "Schema_Name"},
            {
                {"Dev",  "Dev_Server01",  "Dev_DB",  "Dev_Schema"},
                {"Test", "Test_Server01", "Test_DB", "Test_Schema"},
                {"Prod", "Prod_Server01", "Prod_DB", "Prod_Schema"}
            }
        ),

    _DataType = Table.TransformColumnTypes(
        _Env_Table,
        {
            {"Source_Env", type text}, {"Server_Name", type text},
            {"DB_Name", type text}, {"Schema_Name", type text}
        }
    )
in
    _DataType


Step 2: Define a Query Parameter for Dynamic Environment Selection
We need to define a list parameter as p_Query_Environ that controls which environment (Dev, Test, or Prod) Power BI should use when running queries. 
This parameter acts as a switch, allowing you to easily toggle between environments without editing the query code.


Step 3: Pass Connection Strings Dynamically to Queries
Instead of hard‑coding server, database, and schema names, we will filter the Environment Variables Table using the parameter (p_Query_Environ) and feed those values directly into the query connection. 
This makes our queries flexible and environment‑aware, ensuring that Power BI automatically connects to the right source based on the environment we select.

Example: dim_Country

let
    /* 
    Filter the Info_Query_Environment table based on Parameter Value (Eg. Dev, Test, Prod)
    */
    SelectedEnv = 
        Table.SelectRows(Info_Query_Environment, each [Source_Env] = p_Query_Environ){0},

    /* Assign the values to Environment Variables */
    _ServerName   = SelectedEnv[Server_Name],
    _DatabaseName = SelectedEnv[DB_Name],
    _SchemaName   = SelectedEnv[Schema_Name],

    /* SQL Query with dynamic Environment Variables */
    vSrc_Qry = "Select Distinct [Country_ID], [Country_Name], [Currency_Code]
         From " & _SchemaName & ".dim_Country",

    /* Run the SQL Query */
    Source = Sql.Database(_ServerName, _DatabaseName, [Query = vSrc_Qry])
in
    Source

Final Step (Optional): Save Report as Power BI Template
We can save your report as a Power BI Template (.PBIT), which captures the report structure, queries, and parameters, but leaves out the actual data. This makes it ideal for scenarios where different users need to run the same report against different environments (Dev, Test, Prod).


Provide the description to the Report Template while saving.


When a user opens the Power BI Template, it automatically prompts to choose a value for the p_Query_Environ parameter. The options (e.g., Dev, Test, Prod) come directly from the parameter definition that we created earlier.
Based on the selection, Power Query filters the environment table and connects to the correct server, database, and schema for that environment. 
Once the environment is chosen, the user simply clicks Load to run the report, and Power BI retrieves data from the selected environment seamlessly.


Note:
You can refine this method to fit your own scenario. Adjust the setup as needed for your environment.

--------------------------------------------------------------------------------------------------------
Thanks
--------------------------------------------------------------------------------------------------------

Sunday, November 2, 2025

How to Analyze Data Pipeline Run Statistics Using DAX in Power BI

How to Analyze Data Pipeline Run Statistics Using DAX and M-Query
In modern data systems, pipelines are essential for automating the flow of information across platforms and regions. They often run multiple times a day with mixed results, some successful, others failing. To monitor performance effectively, especially to catch recent data load failures, it's important to isolate the latest run for each pipeline. Without this focus, historical logs can become overwhelming and difficult to interpret.

This article explores how to solve that challenge using Power BI. By combining Power Query and DAX, you can flag the latest run, calculate run durations, and extract key metrics like run status and data load success. This approach helps build a clear, dynamic view of pipeline health for better monitoring and decision-making.

Scenario:
Suppose we have a dataset that captures the execution history of data pipelines across various regions. Each record in the dataset represents a single pipeline run and includes columns such as:
  • Pipeline_ID: Unique identifier for each pipeline
  • Pipeline_Name: Descriptive name of the pipeline
  • Pipeline_Region: Deployment region (e.g., US-East, EU-Central)
  • Pipeline_Run_Status: Outcome of the run (Success or Fail)
  • Data_Load_Status: Whether the data load was successful or not.
  • Pipeline_Run_StartDate: Timestamp when the Pipeline run started.
  • Pipeline_Run_EndDate: Timestamp when the Pipeline run completed.

Now we will use this Pipeline Run log data to analyze it further by identifying the latest run for each pipeline, calculating how long each run took, and evaluating whether the data load was successful or not.

Pipeline_Run_Hours:  We can create this Column in Power Query to calculate the Time (in Hours) taken by each Pipeline to run.

let
    _start = [Pipeline_Run_StartDate],
    _end = [Pipeline_Run_EndDate],
    _duration = _end - _start,
    _days = Duration.Days(_duration),
    _hours = Duration.Hours(_duration),
    _minutes = Duration.Minutes(_duration),
    _totalMinutes = (_days * 1440) + (_hours * 60) + _minutes,
    _totalHours = _totalMinutes / 60
in
    _totalHours
--------------------------------------------------------------------------------------------------------

The complete Power Query logic of the above Sample data (Changes dynamically based on the current date time) is as follows:

let
    _today = DateTime.LocalNow(),

// Create pipeline run stats data sample
_pipelineData = #table(
        {
            "Pipeline_ID",    
            "Pipeline_Name",
            "Pipeline_Region",
            "Pipeline_Run_Status",
            "Data_Load_Status",
            "Pipeline_Run_StartDate",
            "Pipeline_Run_EndDate"
        },
        {
            // Alpha pipeline run stats with latest run as Success
            {123, "Alpha", "US-East", "Success", "Success", _today - #duration(29, 6, 45, 0), _today - #duration(28, 8, 30, 0)},
            {123, "Alpha", "US-East", "Success", "Fail",    _today - #duration(25, 7, 15, 0), _today - #duration(24, 8, 45, 0)},
            {123, "Alpha", "US-East", "Fail",    "Fail",    _today - #duration(20, 8, 0, 0),  _today - #duration(19, 10, 0, 0)},
            {123, "Alpha", "US-East", "Success", "Success", _today - #duration(1, 9, 0, 0),   _today - #duration(0, 8, 15, 0)},

            // Beta pipeline run stats with latest run as Fail
            {234, "Beta",  "US-West", "Success", "Success", _today - #duration(28, 6, 30, 0), _today - #duration(27, 9, 15, 0)},
            {234, "Beta",  "US-West", "Success", "Fail",    _today - #duration(22, 7, 45, 0), _today - #duration(21, 9, 30, 0)},
            {234, "Beta",  "US-West", "Success", "Fail",    _today - #duration(1, 8, 30, 0),  _today - #duration(0, 9, 0, 0)},

            // Gamma pipeline run stats with latest run as Fail
            {345, "Gamma", "EU-Central", "Success", "Success", _today - #duration(27, 9, 15, 0), _today - #duration(26, 10, 0, 0)},
            {345, "Gamma", "EU-Central", "Success", "Fail",    _today - #duration(21, 8, 0, 0),  _today - #duration(20, 10, 30, 0)},
            {345, "Gamma", "EU-Central", "Success", "Fail",    _today - #duration(1, 7, 0, 0),   _today - #duration(0, 9, 45, 0)},

            // Delta pipeline run stats with latest run as Success
            {456, "Delta", "Asia-Pacific", "Success", "Success", _today - #duration(30, 6, 30, 0), _today - #duration(29, 7, 45, 0)},
            {456, "Delta", "Asia-Pacific", "Success", "Fail",    _today - #duration(18, 7, 30, 0), _today - #duration(17, 8, 0, 0)},
            {456, "Delta", "Asia-Pacific", "Success", "Success", _today - #duration(2, 8, 15, 0),  _today - #duration(1, 7, 30, 0)},

            // Omega pipeline run stats with latest run as Success
            {567, "Omega", "India", "Success", "Success", _today - #duration(26, 9, 30, 0), _today - #duration(25, 10, 15, 0)},
            {567, "Omega", "India", "Fail",    "Fail",    _today - #duration(15, 8, 30, 0), _today - #duration(14, 10, 30, 0)},
            {567, "Omega", "India", "Success", "Success", _today - #duration(5, 7, 45, 0),  _today - #duration(4, 10, 0, 0)}
        }
    ),

// Add Is_Pipeline_Latest_Run
_addLatestFlag = Table.AddColumn(
        _pipelineData,
        "Is_Pipeline_Latest_Run",
        each 
            let
                _currentPipeline = [Pipeline_ID],
                _currentEndDate = [Pipeline_Run_EndDate],
                _maxEndDate = List.Max(
                    Table.SelectRows(_pipelineData, each [Pipeline_ID] = _currentPipeline)[Pipeline_Run_EndDate]
                )
            in
                _currentEndDate = _maxEndDate,
        type logical
    ),

// Change types
_changeType = Table.TransformColumnTypes(
        _addLatestFlag,
        {
            {"Pipeline_ID", Int64.Type},
            {"Pipeline_Name", type text},
            {"Pipeline_Region", type text},
            {"Pipeline_Run_Status", type text},
            {"Data_Load_Status", type text},
            {"Pipeline_Run_StartDate", type datetime},
            {"Pipeline_Run_EndDate", type datetime},
            {"Is_Pipeline_Latest_Run", type logical}
        }
    ),

// Add Pipeline_Run_Hours
_addRunHours = Table.AddColumn(
        _changeType,
        "Pipeline_Run_Hours",
        each 
            let
                _start = [Pipeline_Run_StartDate],
                _end = [Pipeline_Run_EndDate],
                _duration = _end - _start,
                _days = Duration.Days(_duration),
                _hours = Duration.Hours(_duration),
                _minutes = Duration.Minutes(_duration),
                _totalMinutes = (_days * 1440) + (_hours * 60) + _minutes,
                _totalHours = _totalMinutes / 60
            in
                _totalHours,
        type number
    ),

// Add Pipeline Run Day of Week
_addDayOfWeek = Table.AddColumn(
        _addRunHours,
        "Pipeline_Run_DayOfWeek",
        each Date.DayOfWeekName([Pipeline_Run_StartDate]),
        type text
    ),

// Add Pipeline Duration Category
_addDurationCategory = Table.AddColumn(
        _addDayOfWeek,
        "Pipeline_Run_Duration_Category",
        each 
            let h = [Pipeline_Run_Hours]
            in 
                if h < 2 then "Short" 
                else if h < 6 then "Medium" 
                else "Long",
        type text
    )
in
    _addDurationCategory
--------------------------------------------------------------------------------------------------------

We define the following calculated Column in Power BI which will return TRUE for the latest Run date of each Pipeline (either Success or Fail), otherwise it returns FALSE.

Is_Latest_Pipeline_Run =
VAR CurrentPipeline = tbl_Pipeline_Run_Stats[Pipeline_Id]
VAR CurrentEndDate = tbl_Pipeline_Run_Stats[Pipeline_Run_EndDate]
VAR MaxEndDate =
    CALCULATE(
        MAX(tbl_Pipeline_Run_Stats[Pipeline_Run_EndDate]),
        FILTER(tbl_Pipeline_Run_Stats, tbl_Pipeline_Run_Stats[Pipeline_Id] = CurrentPipeline)
    )
RETURN
    CurrentEndDate = MaxEndDate
--------------------------------------------------------------------------------------------------------
Next, we can define the following set of Measures needed for the Analysis:

Pipeline Run Hours = SUM(tbl_Pipeline_Run_Stats[Pipeline_Run_Hours])

Pipeline Latest Run Hours =
VAR _LatestRunEndDate =
    CALCULATE(
        MAX(tbl_Pipeline_Run_Stats[Pipeline_Run_EndDate]),
        ALLEXCEPT(tbl_Pipeline_Run_Stats, tbl_Pipeline_Run_Stats[Pipeline_Id])
    )
RETURN
    CALCULATE(
        SUM(tbl_Pipeline_Run_Stats[Pipeline_Run_Hours]),
        tbl_Pipeline_Run_Stats[Pipeline_Run_EndDate] = _LatestRunEndDate
    )

Pipeline Latest Run Date =
VAR _CurrentPipeline = MAX(tbl_Pipeline_Run_Stats[Pipeline_Id])
VAR _CurRunEndDate = MAX(tbl_Pipeline_Run_Stats[Pipeline_Run_EndDate])
VAR _LatestRunEndDate =
    CALCULATE(
        MAX(tbl_Pipeline_Run_Stats[Pipeline_Run_EndDate]),
        ALLEXCEPT(tbl_Pipeline_Run_Stats, tbl_Pipeline_Run_Stats[Pipeline_Id])
    )
RETURN
    IF(_CurRunEndDate = _LatestRunEndDate, _LatestRunEndDate)

Latest Pipeline Run Status =
VAR _CurrentPipeline = MAX(tbl_Pipeline_Run_Stats[Pipeline_Id])
VAR _LatestRunEndDate =
    CALCULATE(
        MAX(tbl_Pipeline_Run_Stats[Pipeline_Run_EndDate]),
        ALLEXCEPT(tbl_Pipeline_Run_Stats, tbl_Pipeline_Run_Stats[Pipeline_Id])
    )
RETURN
    CALCULATE(
        MAX(tbl_Pipeline_Run_Stats[Pipeline_Run_Status]),
        tbl_Pipeline_Run_Stats[Pipeline_Id] = _CurrentPipeline &&
        tbl_Pipeline_Run_Stats[Pipeline_Run_EndDate] = _LatestRunEndDate
    )

Latest Data Load Status =
VAR _CurrentPipeline = MAX(tbl_Pipeline_Run_Stats[Pipeline_Id])
VAR _LatestRunEndDate =
    CALCULATE(
        MAX(tbl_Pipeline_Run_Stats[Pipeline_Run_EndDate]),
        ALLEXCEPT(tbl_Pipeline_Run_Stats, tbl_Pipeline_Run_Stats[Pipeline_Id])
    )
RETURN
    CALCULATE(
        MAX(tbl_Pipeline_Run_Stats[Data_Load_Status]),
        tbl_Pipeline_Run_Stats[Pipeline_Id] = _CurrentPipeline &&
        tbl_Pipeline_Run_Stats[Pipeline_Run_EndDate] = _LatestRunEndDate
    )

The following measure is based on the column Pipeline_Run_EndDate to identify the latest run Date of the Pipeline. It returns 1 for latest run date and 0 for others.
This measure logic and objective is similar to the calculated column Is_Latest_Pipeline_Run created in the beginning.
However, this will not carry forward in Drill through filters like Is_Latest_Pipeline_Run.

Pipeline Latest Run Flag =
VAR _CurRunEndDate = MAX(tbl_Pipeline_Run_Stats[Pipeline_Run_EndDate])
VAR _LatestRunEndDate =
    CALCULATE(
        MAX(tbl_Pipeline_Run_Stats[Pipeline_Run_EndDate]),
        ALLEXCEPT(tbl_Pipeline_Run_Stats, tbl_Pipeline_Run_Stats[Pipeline_Id])
    )
RETURN
    IF(_CurRunEndDate = _LatestRunEndDate, 1, 0)

Results:
Now we can generate the Pipeline Latest Run Summary table using the above measures as shown below:
You can apply required conditional formatting for the Latest Data Load Status if needed.


The Detailed table view of the Pipeline and Data Load Status details as per below:


Please Note:
To establish a relationship between the Pipeline Run Statistics table and the Calendar table based on date, please convert either Pipeline_Run_StartDate or Pipeline_Run_EndDate to Date format. 
Power BI does not support relationships between columns of different data types, specifically, between Date and DateTime columns.
--------------------------------------------------------------------------------------------------------
Drill through from Summary Page to Details Page:
Let's say, you have Summary table in the Power BI page "Pipeline_Run_Summary" and the details table view in the other Page "Pipeline_Run_Details".

Now if you want to Drill through from Summary page to Details page, then make sure there must be one common Column (Eg. Piplie_Id, Pipeline_Region) selected in both Visuals.

Next add the required common Fields in the Drill Through targe page (Pipeline_Run_Details), and add the additional columns if needed as shown below:


Next go back to the Summary page and try drill though from a Record from the Visual:


Now the result of the drill through in Details page is as shown below, will the all the Filters apply from Summary to Details page, except Latest Run Date filter is not passed:



Why Drill Through is not Filtering the Latest Dates in Detail view:
This issue arises as the measures (Eg. Pipeline Latest Run Date) used in the Summary view are calculating the values based on internal logic that determines the latest run date for each pipeline using DAX. 
  • Implicit filters inside measures (like MAX, CALCULATE, or ALLEXCEPT) are local to the measure and don’t become part of the drillthrough filter context.
  • Drillthrough only carries filters from explicit fields used in the visual or slicer like actual columns or selected values.
  • Measures don’t expose their internal logic to the drillthrough engine.
So, when you drill through from a visual that displays the latest run date, Power BI doesn’t automatically pass that date as a filter. It’s simply the output of a measure, not a selected or filtered field. To enable drill through filtering, you need to use actual columns or calculated columns that are part of the model and can participate in the filter context.

Enable Drill Through to pass Latest Date Filter from Summary to Detail view:
To enable filtering by the latest Pipeline Run Date during drill-through, we can apply a Visual Level or Page Level filter using the Is_Latest_Pipeline_Run column in the Summary View. This ensures that only the most recent run for each pipeline is passed to the details page during drill-through.

Output:
Now, the drill-through results on the Details page will correctly reflect the latest pipeline run date filter, as applied using the Is_Latest_Pipeline_Run column.



--------------------------------------------------------------------------------------------------------
Thanks
--------------------------------------------------------------------------------------------------------

Featured Post from this Blog

How to compare Current Snapshot Data with Previous Snapshot in Power BI

How to Dynamically compare two Snapshots Data in Power BI Scenario: Suppose we have a sample Sales data, which is stored with Monthly Snapsh...

Popular Posts from this Blog