Improve inputs for buildings retrofit calculations

[ekatef]

Changes proposed in this Pull Request

The PR is intended to update building stock data used in calculations for retrofitting costs. The following changes are suggested.

1. Integrate updated hotmaps data taken from gitlab repo.
2. Removes a patch for Poland, which may be replaced by a recent fix of data structure: the problem for Poland was caused by using a singular form for Construction features (U-value) instead of plural.

Checklist

• I tested my contribution locally and it seems to work fine.
• Code and workflow changes are sufficiently documented.
• Changed dependencies are added to envs/environment.yaml.
• Changes in configuration options are added in all of config.default.yaml.
• Changes in configuration options are also documented in doc/configtables/*.csv.
• A release note doc/release_notes.rst is added.

[ekatef]

Not sure about a proper strategy to add the updated hotmaps dataset to the repo.

Data inputs for retrofitting are committed into data/retro/ and we could add the updated hotmaps there, but currently data folder is git-ignored. Which probably implies that this way is not a preferable one.

Apart of that, not sure it we want to keep the previous version of hotmaps dataset, or replace it with an updated version.

@martacki what is you vision of the points above?

[ekatef]

The PR was put on hold for quite a while due to some anomalies found in the response on the data update. After digging into the data, it has been found that the anomalies are linked with the structure of the data updates itself.

The PR is ready for review, but may need some additional local testing to ensure that it's fully consistent with all the latest changes.

Update: the local testing successful with an additional pandas-related fix.

@martacki please, feel free to review, when you have time 🙂

[ekatef]

Update: added a number of additional maintenance fixes to address recent pandas deprecations:

• re-implemented a previous fix for axis=1 deprecation in groupby( ) to avoid breaking the data workflow;
• fix a chained assignment warning triggered by inplace in rename( );
• fix some stack deprecation warnings by setting future_stack=True, except one case when we are using dropna=False. I don't quite get how should stack behaviour change in the future version, but it would for sure effect treatment of NaNs, so it feels risky to give a consent for these changes.

[ekatef]

As a comment, these updates of hotmaps data are in fact application of some country-specific scaling coefficients. That is an example of how data structure looks like:

The testing has confirmed that these changes are accurately transferred to the building stock data in the model, and effect mainly the overall area data floor_area_elec_s_{n_clusters}.csv.

[martacki]

Hi @ekatef I tried to evaluate your implementation, comparing the resulting non-annualised costs both from your implementation as well as from the master branch (that's why I merged master).
Also, I compared the resulting p_nom_max values for retrofitting. Here are the results:

comparing costs, there are small differences of below 1 EUR/m2 for most countries, which I'd ignore. 1-2%, seems reasonable.

Only for PL, the results are quite heavy, but only for costs, not for the potential dE savings:
For the residential sector, the costs are exactly 50% compared to the current master branch, for services they're exactly 2/3 and the total costs are 65% of the costs for the master branch (moderate scenario).

For the ambitious scenario, the residential is also exactly 50% of what's in current master, but the costs for the services sector is also exactly 50%. Therefore, total is also 50%. I wonder what is the impact on Poland?

When looking on the potential (how much "MW" of retrofitting can be installed), the update to new HOTMAPS does have a larger impact (positive means that the new potential is smaller than before):


But given that the differences are also below 3%, I'd say that that's fine.

The generation profile is exactly the same in both branches (p_min_pu and p_max_pu)

I'd say this looks good, it would be just useful to understand the impact on PL, mainly on the costs. I understand that you're suggesting to use the HOTMAPS data for PL, instead of Eurostat data. Maybe this is where the huge difference comes from? Would be good to know which of these sources is more accurate.

[martacki]

Had also a look into the capital_costs of the prepared network (not the EUR/m2, but the resulting costs after multiplying the costs per m2 with the floor area), and here's the result - which will have the biggest impact (annualised costs) on the results: (positive means the costs are now lower, negative numbers mean they're higher).

looks in line with the plot you posted, however I'm a bit worried about the tremendous increase of costs with the new update, retrofitting will be 60-100% more expensive in approx. half of all the countries...

[fneum]

Let me know when this should be reviewed to go into the master. It would be good to have a rough sense for where the cost differences come from.

The added input file should definitely not be part of the git repository.

I would switch to a remote input and also purge the file from the git history:

Purging

pip install git-filter-repo
git filter-repo --path data/retro/data_building_stock.csv --invert-paths
git push origin --force --all

Remote Input

rule ...:
    input:
        building_stock=storage(
            "https://gitlab.com/hotmaps/building-stock/-/raw/master/data/building_stock.csv",
            keep_local=True,
        ),

[fneum]

@ekatef @martacki where did we land here?

[martacki]

I think we stopped investigating. Unless @ekatef plans to pick this up, we can close

[ekatef]

Hello!
@fneum apologies for letting the PR hanging for quite a while.

The major result of our investigation is that the update of the building stock area A may be not quite consistent with the overall heating demand E. The point is E and A are connected via the energy intensity E/A of buildings which must be more or less invariant (quality of the building stock hasn't been changed due to the fact that estimates for the number of buildings were incomplete). However, we are using independent datasets for E and A, and it's a bit tricky to trace the status of both and as @martacki has mentioned we didn't have resources to do so. So, we have ended up by using the updated building stock data only for post-processing.

Closing this PR as it feels like some additional work is advisable to understand all the implications for modeling results. To get some value from these efforts, I'll try to review the content of the changes and contribute a part which relates to bug-fixing as a new PR

@fneum let us know please if you have any comments on that!