Data Intelligence Accelerates Naturgy’s Transformation.

After 180 years in the energy sector, Naturgy still manages to provide smart and innovative solutions by staying at the forefront of technology. Over the years, it has diversified its business, becoming the first company that markets both gas and electricity in Spain.
In the past few years, Naturgy has made decisive changes to its strategy, expanding its international reach and pioneering clean energies, new energy transition technologies and digitisation.

Using this love of innovation as a guiding principle, we at Paradigma have helped Naturgy to develop a cutting-edge product whose main goals are as follows: Integrating energy and market supply and management data intelligence; and managing the technical coordination of the negotiations of the financial conditions of gas supply contracts.

Data analysis to improve the decision-making process.

The main challenge the Naturgy team faces is uploading and managing all data sources that it needs to perform analyses that will allow it to make the right decisions.
Before, the many data sources were manually downloaded and processed in an Excel spreadsheet on a recurring basis, taking up most of the Business team’s time. Moreover, these data were worked on in a local environment, which gave rise to a strong dependence on specific people within the team to keep the entire project chugging along.

This project is the result of three phases (a pilot project and two implementation stages) where Naturgy trusted in Paradigma to overcome the challenges they had set for themselves.

All data used in this area are collected from different free-access and conditional-access websites. Originally, the Naturgy team spent some time every week visiting and browsing websites, downloading the necessary files and processing them in a spreadsheet that was used as a decision-making dashboard.

This had several drawbacks:

• A member of the team downloaded a file into their computer and added the data to the spreadsheet, which made it impossible for all members to have the same information at all times.
• This manual, repetitive downloading of files took up much of the team’s time.

To improve this process, we developed a solution for automating all file downloads in a cloud environment, ensuring scalability and shared data access. The dashboard was moved to a display tool that will allow the team to focus on analysing the data rather than on processing them.

The project was divided into the following stages:

• Identification of all data sources Naturgy used and how it obtained the data.
• Definition of an architecture solution in the cloud, specifically in Azure.
• Automation of the downloading of the data and its storage in Azure in a first raw layer.
• Processing of the data to generate a silver layer, which allows the information to be analysed.
• Development in Power BI of the information analysis dashboard.

The project is composed of the following:

The file downloading has been automated with Azure Functions, where scraping processes are periodically run with Python to trawl the different websites for new files.

The downloaded files are stored in Azure Blob Storage so that the Naturgy team can access them and check them in their raw layer.

The files in the raw/silver layer are also stored in Azure Data Lake in order to centralise the information of the data at Naturgy and make all the content available so that it may be used by the entire company.

In a last data processing step, the information to be analysed is written in Azure SQL. Finally, the Azure SQL data are the source for the dashboard in Power BI.

Midway through the project we detected that, even though we had moved the solution to Azure, it was possible to go one step beyond and add a basic first CI/CD cycle that would allow the Naturgy team to have both a development environment and a production environment together with an agile management of all changes to the process.

The particularity of where data are gleaned from is particularly important in this project, since the sources in many cases are constantly changing public websites.

Due to them being subjected to constant changes, small adjustments will have to be made in the data collection code, so having a CI/CD cycle that enables the team to easily upload the changes to production is particularly important.

This project has laid the groundwork for future data projects at Naturgy, which may nudge the company to evolve toward data management, processing and storage so as to get a much better return therefrom.

Paradigma worked on the entire project very closely to Naturgy by forming a single team. Thus, it was possible to iterate the solution from an agile point of view. Given that we developed this project 100% remotely, the Naturgy team’s availability was essential to successfully arrive at a solution.

This fluid and constant communication enabled us to match the solution closely to Naturgy’s use and needs, besides facilitating the transfer of knowledge, it being a tool that the company could use from the very beginning.

Thanks to the solution we proposed, the Naturgy team can spend most of its time analysing data in a centralised, available display tool.

We have developed a dashboard that allows the team to devote its efforts to monitoring and analysing a series of variables so as to be able to take appropriate action when negotiating new conditions in gas supply contracts.

The starting situation has been improved:

• By moving everything to a centralised cloud environment, the dependence on specific people has decreased. The entire team has at all times a single valid version of the data and the dashboard, so everybody can work on a common truth.
• All repetitive manual work has been eliminated, so team members can concentrate on what is really important.

Improved decision-making.

As a result, we expect the solution to improve Naturgy’s decision-making process and allow the team to better negotiate gas supplies, thus achieving a better return.

Additionally, we laid down a series of data project-related good practices. Therefore, the foundations have been laid for other Naturgy departments to evolve, which can always count on us to help them to improve their processes.