Innovating infrastructure for the new smart meter network

When Norway introduced compulsory smart meters, Hafslund, the country’s largest power company, met the new, complex data challenge with a solution built on Google BigQuery, Google Cloud Bigtable, and Google Cloud Platform.

Hafslund is not only the largest power company in Norway, it also owns the country’s largest electricity network, Hafslund Nett, connecting more than 700,000 customers. By using technological innovation to optimize efficiency, Hafslund makes sure it keeps maintenance call-outs to a minimum and costs down. Now the company faces new challenges, thanks to developments in regulations, which have far-reaching implications.

“The whole business is changing,” says Jon Andreas Pretorius, CIO at Hafslund Nett. “New regulations in Norway require smart meters in every household. The industry has to tackle new data demands, new challenges, and new possibilities. Norway is acting now, but this transition will happen in every EU country. Within a few years, every home will have an automatic meter sending electricity meter readings every hour to a central hub.”

That transition creates multiple data and infrastructure challenges. Hafslund has to work with a much greater quantity of data than before, as well as new data types that did not previously exist. The company has to devise storage, usage, and transmission solutions ahead of competitors while improving customer service and creating new efficiencies.

At the same time, the new smart meter infrastructure operates through a large radio communication network, requiring maintenance and monitoring. To track, analyze, and use all of this new data, Hafslund looked to create a scalable, reliable solution that leveraged managed services to minimize maintenance burden.

For core processes, if we can buy something that matches what we need 100%, then it makes more sense to buy it than make it ourselves. Tech is one of the main ways in which we compete, by creating efficiencies from preemptive maintenance, so we don’t send staff into the field unnecessarily.

Jon Andreas Pretorius, CIO, Hafslund Nett AS

A simple solution to a complex data challenge

By 2020, there will be smart meters in 80% of EU households, according to the European Commission. Smart meters deliver automated electricity readings every hour to generate savings for consumers and major energy efficiencies across the grid. Getting ahead of the change, Hafslund has already deployed smart meters to almost 80% of its 710,000 customers, working with electricity vendors and new grid infrastructure to integrate the new devices.

“To send us meter readings data, electricity vendors in Norway deliver XML files of all of their meter data once an hour,” says Jon. “We are working with them to stream that data instead, and a large part of our challenge is to convince vendors to step into the modern world and embrace streaming, real-time processing, and an ETL model.”

The new data does not just relate to electricity readings. “The Norwegian grid is a bit unusual,” says Jon. “In Norway, radio is the most viable means of communicating meter readings to a central hub. As a consequence, we also handle diagnostics data from the meters related to their mesh network. We used to be a major utilities company. Now we are a major utilities company and a radio network operator. That came as a surprise to everybody.”

The company looked to solve both challenges with a scalable data warehouse able to import, process, and analyze meter data in real-time. To do that, Hafslund approached IT consultancy Computas, with which it has a long-standing relationship. “Computas is and has been a strategic partner for Hafslund in transforming and modernizing our IT architecture” says Jon. “Computas has therefore been instrumental in Hafslund’s cloud strategy, providing their expertise and experience on the Google Cloud Platform.”

Together, the teams examined potential solutions. “We looked at a cloud-based data warehouse that worked well once the data was stored, but getting data into it was very difficult,” remembers Jon. “We need to work with a constant stream of individual small readings, but this solution required that we store data elsewhere and deliver it to the data warehouse every hour with a batch transfer.”

Because all the data and diagnostics from the network are on Google BigQuery, we can pinpoint a problem more easily if a meter isn’t communicating. That means when we send repair staff into the field to fix things, they can make the most efficient and effective use of their time

Jon Andreas Pretorius, CIO, Hafslund Nett AS

That’s why Hafslund chose Google BigQuery on Google Cloud Platform. Because Google BigQuery works directly with streamed data, there is no need for additional storage or hourly batch transfers. “We have all the data in Google BigQuery, which means we can run aggregations across all of the meters at the same time and use structural data, such as the location of the meters,” says Jon. As a managed service, Google BigQuery provides automatic scaling, helping to ensure that the data warehouse delivers performance and reliability.

As well as the electrical meter readings, all diagnostic and structural data goes to Google BigQuery, too, so Hafslund can use it to analyze the health of its radio mesh. “Because all the data and diagnostics from the network are on Google BigQuery, we can pinpoint a problem more easily if a meter isn’t communicating,” says Jon. “That means when we send repair staff into the field to fix things, they can make the most efficient and effective use of their time.”

Lightning-fast transactional storage

To complement the Google BigQuery solution, Hafslund built out into Google Cloud Bigtable for transactional storage that gives customers, staff, and partners responses to queries in microseconds. Data is delivered to both Google BigQuery and Cloud Bigtable through a combination of Google Cloud Pub/Sub and Google Cloud Dataflow, as a reliable, direct way to deliver data without information loss.

“It took us one day to start using Google BigQuery for storage,” says Jon. “It was very, very easy to set up, and has been great to use. It’s worked with no problems and no downtime for all of the time we’ve used it. Even better, it’s evolving. There are newer partitioning possibilities that weren’t in place a year ago and that are really useful to us now.”

Streaming meter data in real-time to customer dashboards

Thanks to the new, granular data solution, Hafslund minimizes unnecessary call-outs to change meters, saving engineers and maintenance workers time. “Instead of sending 10 people out to switch 10 meters, we can see when only one meter is a problem, and send out just 1 person,” says Jon. “As far as I know, the functionality we’ve developed for collecting the diagnostic data to maintain our radio network is something that you just can’t find on the market. It’s new knowledge where before there was nothing.”

Next, Hafslund plans to use Cloud Dataflow to create real-time dashboards for customers, so they can check up-to-date readings of their meters from anywhere, at any time.

Now we can stream data, so we can provide consumption information to customers on their phones without worrying about scaling and performance. We’re getting platforms up, streaming data; and storing records. All that’s taken care of, so we can just work on the service

Jon Andreas Pretorius, CIO, Hafslund Nett AS