Everything You Wanted To Know About Data Export (But Were Afraid to Ask)*

Data Analysis
4

Dealing With Duplicates

When I first looked at the Sense Export, I was baffled to see a huge number of duplicate entries - Two Device entries for the same hour often with different values. Then I realized that device Names can refer to multiple Device IDs, which are the real detections, that have been merged. I then did a count of duplicates by Name and Device ID and got the following list:

   Name                        DevID        count
                                  
 1 AC 3                        4466b6a8       834
 2 AC 3                        90172486       834
 3 Always On                   always_on        2
 4 AppleTV Family Room         da94c832         2
 5 AppleTV Master              caa89d7f         2
 6 Cable Amps                  0b550d1e         2
 7 Canon Printer               E4KDKPsK         1
 8 Canon Printer               ks5w26g4         1
 9 Coffee Maker                35d584f6         8
10 Coffee Maker                b05db37b         8
11 Family Room Switch          a12d3fb8         2
12 Fridge                      f022103b         2
13 Fridge 2                    ebd937ef         2
14 Furnace Down                dtSNTXPF         2
15 Furnace Up                  OsBEZsNy         2
16 HP LaserJet                 3b211685         2
17 Hue Down                    09190c6b         2
18 Keene Vent Bridge           46d8391d         2
19 KP115 Roamer                ssi-78bdfde3     2
20 Kyle’s PC                   da6fd827         2
21 Microwave                   d0956b5a       129
22 Microwave                   e0a3bf5d       129
23 Modem                       6f6e7d08         2
24 Office Always On            1022400f         2
25 Playroom Cluster            ba04ecc2         2
26 Recirculation Pump          9f9090d6         2
27 Router                      4bb72296         2
28 Solar Production            solar            2
29 SolarCity Bridge            80cd62b4         2
30 Switch                      e4372a48         2
31 TiVo                        db039872         2
32 Tivo Mini                   04894581         2
33 Total Usage                 mains            2
...

Surprisingly there appeared to be 2 kinds of duplicates.

  • Same Name but different DevIDs - that’s the merged device case, like AC 3, Coffee Maker and Microwave.
  • Same Name, single DevID - How did those happen ? How, became an apparent when I looked at the date and time of those duplicates, all at 1AM on Nov 6th. All of those are related to the extra “fall back” daylight savings transition hours.

Once I moved all those Nov 6, 1AM duplicates to the extra hour, I had a much shorter list, that made sense due to merges.

  Name          DevID    count
1 AC 3          4466b6a8   834
2 AC 3          90172486   834
3 Canon Printer E4KDKPsK     1
4 Canon Printer ks5w26g4     1
5 Coffee Maker  35d584f6     8
6 Coffee Maker  b05db37b     8
7 Microwave     d0956b5a   129
8 Microwave     e0a3bf5d   129

There’s still one minor mystery - the Canon Printer is not merged and is a smart plug. I suspect this duplicate happened when I was setting up the smart plug and I might have reset the device and connected a second time.

I should also point out that I have a number of smart plugs that are merged using “What’s connected to this ?”, but none of those show as duplicates in the Export. That’s because Sense Export only outputs the data from the smart plug, since it is the smart plug reflects the true data for the whole device. Not the same as merge of native detections that are additive.

1 Like