Incremental Refresh in Power BI, Part 3: Best Practices for Large Semantic Models

Incremental Refresh in Power BI, Best Practices for Large Semantic Models

In the two previous posts of the Incremental Refresh in Power BI series, we have learned what incremental refresh is, how to implement it, and best practices on how to safely publish the semantic model changes to Microsoft Fabric (aka Power BI Service). This post focuses on a couple of more best practices in implementing incremental refresh on large semantic models in Power BI.

Note

Since May 2023 that Microsoft announced Microsoft Fabric for the first time, Power BI is a part of Microsoft Fabric. Hence, we use the term Microsoft Fabric throughout this post to refer to Power BI or Power BI Service.

The Problem

Implementing incremental refresh on Power BI is usually straightforward if we carefully follow the implementation steps. However in some real-world scenarios, following the implementation steps is not enough. In different parts of my latest book, Expert Data Modeling with Power BI, 2’nd Edition, I emphasis the fact that understanding business requirements is the key to every single development project and data modelling is no different. Let me explain it more in the context of incremental data refresh implementation.

Let’s say we followed all the required implementation steps and we also followed the deployment best practices and everything runs pretty good in our development environment; the first data refresh takes longer, we we expected, all the partitions are also created and everything looks fine. So, we deploy the solution to production environment and refresh the semantic model. Our production data source has substantially larger data than the development data source. So the data refresh takes way too long. We wait a couple of hours and leave it to run overnight. The next day we find out that the first refresh failed. Some of the possibilities that lead the first data refresh to fail are Timeout, Out of resources, or Out of memory errors. This can happen regardless of your licensing plan, even on Power BI Premium capacities.

Another issue you may face usually happens during development. Many development teams try to keep their development data source’s size as close as possible to their production data source. And… NO, I am NOT suggesting using the production data source for development. Anyway, you may be tempted to do so. You set one month’s worth of data using the RangeStart and RangeEnd parameters just to find out that the data source actually has hundreds of millions of rows in a month. Now, your PBIX file on your local machine is way too large so you cannot even save it on your local machine.

This post provides some best practices. Some of the practices this post focuses on require implementation. To keep this post at an optimal length, I save the implementations for future posts. With that in mind, let’s begin.

Best Practices

So far, we have scratched the surface of some common challenges that we may face if we do not pay attention to the requirements and the size of the data being loaded into the data model. The good news is that this post explores a couple of good practices to guarantee smoother and more controlled implementation avoiding the data refresh issues as much as possible. Indeed, there might still be cases where we follow all best practices and we still face challenges.

Note

While implementing incremental refresh is available in Power BI Pro semantic models, but the restrictions on parallelism and lack of XMLA endpoint might be a deal breaker in many scenarios. So many of the techniques and best practices discussed in this post require a premium semantic model backed by either Premium Per User (PPU), Power BI Capacity (P/A/EM) or Fabric Capacity.

The next few sections explain some best practices to mitigate the risks of facing difficult challenges down the road.

Practice 1: Investigate the data source in terms of its complexity and size

This one is easy; not really. It is necessary to know what kind of beast we are dealing with. If you have access to the pre-production data source or to the production, it is good to know how much data will be loaded into the semantic model. Let’s say the source table contains 400 million rows of data for the past 2 years. A quick math suggests that on average we will have more than 16 million rows per month. While these are just hypothetical numbers, you may have even larger data sources. So having some data source size and growth estimation is always helpful for taking the next steps more thoroughly.

Practice 2: Keep the date range between the RangeStart and RangeEnd small

Continuing from the previous practice, if we deal with fairly large data sources, then waiting for millions of rows to be loaded into the data model at development time doesn’t make too much sense. So depending on the numbers you get from the previous point, select a date range that is small enough to let you easily continue with your development without needing to wait a long time to load the data into the model with every single change in the Power Query layer. Remember, the date range selected between the RangeStart and RangeEnd does NOT affect the creation of the partition on Microsoft Fabric after publishing. So there wouldn’t be any issues if you chose the values of the RangeStart and RangeEnd to be on the same day or even at the exact same time. One important point to remember is that we cannot change the values of the RangeStart and RangeEnd parameters after publishing the model to Microsoft Fabric.

Continue reading “Incremental Refresh in Power BI, Part 3: Best Practices for Large Semantic Models”

Incremental Refresh in Power BI, Part 2; Best Practice; Do NOT Publish Data Model Changes from Power BI Desktop

Incremental Refresh Best Practice, Do NOT Publish Changes from Power BI Desktop

In a previous post, I shared a comprehensive guide on implementing Incremental Data Refresh in Power BI Desktop. We covered essential concepts such as truncation and load versus incremental load, understanding historical and incremental ranges, and the significant benefits of adopting incremental refresh for large tables. If you missed that post, I highly recommend giving it a read to get a solid foundation on the topic.

Now, let’s dive into Part 2 of this series where we will explore tips and tricks for implementing Incremental Data Refresh in more complex scenarios. This blog follows up on the insights provided in the first part, offering a deeper understanding of how Incremental Data Refresh works in Power BI. Whether you’re a seasoned Power BI user or just getting started, this post will provide valuable information on optimising your data refresh strategies. So, let’s begin.

When we publish a Power BI solution from Power BI Desktop to Fabric Service, we upload the data model, queries, reports, and the loaded data into the data model to the cloud. In essence, the Power Query queries, the data model and the loaded data will turn to the Semantic Model and the report will be a new report connected to the semantic model with Connect Live storage mode to the semantic model. If you are not sure what Connect Live means, then check out this post where I explain the differences between Connect Live and Direct Query storage modes.

The Publish process in Power BI Desktop makes absolute sense in the majority of Power BI developments. While Power BI Desktop is the predominant development tool to implement Power BI solutions, the publishing process is still not quite up to the task, especially on more complex scenarios such as having Incremental Data Refresh configured on one or more tables. Here is why.

As explained in this post, publishing the solution into the service for the first time does not create the partitions required for the incremental refresh. The partitions will be created after the first time we refresh the semantic model from the Fabric Service. Imagine the case where we successfully refreshed the semantic model, but we need to modify the solution in Power BI Desktop and republish the changes to the service. That’s where things get more complex than expected. Whenever we republish the new version from Power BI Desktop to Fabric Service, we get a warning that the semantic model exists in the target workspace and that we want to Overwrite it with the new one. In other words, Power BI Desktop currently does not offer to apply the semantic model changes without overwriting the entire model. This means that if we move forward, as the warning message suggests, we replace the existing semantic model and the created partitions with the new one without any partitions. So the new semantic model is now in its very first stage and the partitions of the table(s) with incremental refresh are gone. Of course, the partitions will be created during the next refresh, but this is not efficient and realistically totally unacceptable in production environments. That’s why we MUST NOT use Power BI Desktop for republishing an already published semantic model to avoid overriding the already created tables’ partitions. Now that Power BI Desktop does not support more advanced publishing scenarios such as detecting the existing partitions created by the incremental refresh process, let’s discuss our other options.

Alternatives to Power BI Desktop to Publish Changes to Fabric Service

While we should not publish the changes from Power BI Desktop to the Service, we can still use it as our development tool and publish the changes using third-party tools, thanks to the External Tools support feature. The following subsections explain using two tools that I believe are the best.

Continue reading “Incremental Refresh in Power BI, Part 2; Best Practice; Do NOT Publish Data Model Changes from Power BI Desktop”

Microsoft Fabric: Use Copilot to Generate Data Model Synonyms

Microsoft Fabric: Use Copilot to Generate Data Model Synonyms

One of my older posts explains how to enable Copilot on Fabric and how to use Copilot to generate Power BI reports. In this post, I aim to explain yet another use case for Copilot that can help us to make a better and more useful semantic model in Power BI using synonyms. In an old post published in May 2016, I explained how to use Power BI synonyms to take our Power BI Q&A experience to another level. In that post, I explained how we could use synonyms to translate data model objects in different languages so the end-user could ask questions in their native language and get the results in Power BI. That was such a cool use case for synonyms, I suppose, wasn’t it? Fast track to December 2023, I believe the Q&A is still one of the coolest Power BI features that stands out when demoing the solutions to the customers; therefore, it makes absolute sense to use synonyms to improve the Q&A‘s efficiency and accuracy. This blog post explores the possibility of using Copilot to define synonyms in Power BI Desktop.

Prerequisites

As explained here, we first need to enable Copilot on Fabric Service. Please note that the technique explained in this post requires either Power BI Premium Capacity or at least F64 Fabric capacity and won’t work on PPU, Embedded capacities, Fabric capacities smaller than F64 or Fabric Trial (FT) capacities.

We also need to have the latest version of Power BI Desktop installed on our machine. With that, let’s begin.

Using Power BI Copilot to generate synonyms

While defining synonyms for the semantic model objects significantly helps with the Q&A experience, it is still a cumbersome process if done manually. So, if we meet the prerequisites, we can summon Copilot to the rescue. Follow these steps after opening a Power BI file in Power BI Desktop:

  1. Ensure you’re signed into Fabric service with your account
  2. Click the Insert tab
  3. Select the Q&A visual
  4. On the Q&A visual, click the Q&A Setup button shown with a gear icon
  5. On the Q&A Setup window, you must see a message offering to “Improve Q&A with synonyms from Copilot” on top of the window; click the Add synonyms button

The following image shows the preceding steps:

Improve Q&A with synonyms from Power BI Copilot in Microsoft Fabric
Improve Q&A with synonyms from Copilot
Continue reading “Microsoft Fabric: Use Copilot to Generate Data Model Synonyms”

Quick Tips: Find Power BI Desktop Local Port Number with Model Explorer

Quick Tips: Find Power BI Desktop Local Port Number with Model Explorer

In March 2018, I wrote a blogpost called Four Different Ways to Find Your Power BI Desktop Local Port Number. Last week, Zoe Doughlas from Microsoft left a comment reminding me of a fifth method to get the port which encouraged me to write this quick tip. Thanks to Zoe!

As the name suggests, the blog was about finding Power BI Desktop’s local port number. If you do not have any clue what I mean by local port number, I strongly suggest reading that blog.

This blog focuses on yet another method that wasn’t available back then. Indeed, it is a new feature added to the October 2023 release of Power BI Desktop. This is a Quick Tip so let’s jump straight to the topic and learn how we find the port number (and more) in Power BI Desktop (Oct 2023 and later releases).

Prerequisites

As mentioned, this new feature was added to Power BI Desktop’s October 2023; therefore, we must install that release on our local machine. Indeed, the October 2023 release was packed with many other features, including the Model Explorer (the topic of this blog) and the ability to define calculation groups directly in Power BI Desktop. Many of these features are still in preview; hence, they require enabling.

The following few steps explain how to enable Preview Features in Power BI Desktop:

  1. Open Power BI Desktop and click Settings (the gear icon) from the right pane
  2. On the Options page, from the GLOBAL section, click the Preview features tab
  3. Enable the desired features; for this blog, we need the Model explorer and Calculation group authoring
  4. Click OK

The following image shows the above steps:

Enabling Preview Features in Power BI Desktop
Enabling Preview Features in Power BI Desktop

Depending on the selected features, you may need to restart your Power BI Desktop to allow them to enable.

Looking at the above image, some of you may ask “Soheil, are you using an older version of Power BI Desktop?” and I am glad you asked. The answer as always is “It depends”. And, this time it depends on the timing of writing this blog which is early December 2023, and the fact that Power BI Desktop November 2023 was released a couple of weeks ago, therefore, Power BI Desktop October 2023 is kind of OLD! And, YES! I installed Power BI Desktop Nov 2023 for the sake of writing this blogpost.

Continue reading “Quick Tips: Find Power BI Desktop Local Port Number with Model Explorer”