I have a lot of data, Gbs so I only used it with a sample of Workout data. I agree standardizing column name across data sources would be useful. The Apple Health data also has a variety of attributes based on category so I’m not sure how well the correlation would work across all variables. I’ve not tried it yet. Maybe some logic needs to be added? I really like the heatmap grid though.
Depending on how many columns, and particularly if there null values, the choice is to remove them. It might make sense to make smaller data frames to analyze just what you want to see. Without divulging anything, you could share what the Apple columns are, and maybe even a snapshot. Not sure if I can see a way through, but willing to try.
For myself, I’ve just been toying with graphs and averages, so downloaded a new data set, and will eventually go back and do more data cleansing. I imagine it will become more useful over time, as I accumulate data points.
Yes, I’m only looking at Workout data so just a sample of the Health data. Of course, displaying workouts, BGs, etc., in one graph is the goal so I can view the overall effect is on BGs. I’m not that familiar with the plot routines but I will keep looking into it.
For example, sample workout record from Workout.csv:
“Apple Watch”,“6.2.8”,"<<HKDevice: 0x283f81ae0>, name:Apple Watch, manufacturer:Apple Inc., model:Watch, hardware:Watch4,2, software:6.2.8>",2020-09-08 18:43:37 -0400,2020-09-08 18:17:06 -0400,2020-09-08 18:43:36 -0400,“HKWorkoutActivityTypeWalking”,26.50305813352267,“min”,1.202586463674468,“mi”,71.3755684196788,“kcal”
And the heatmap in your python script generates for all data in Workout.csv:
“Loop”,“57”,"<<HKDevice: 0x283eb1ef0>, name:CGMBLEKit, manufacturer:Dexcom, model:G6, software:21.0, UDIDeviceIdentifier:00386270000385>",“BloodGlucose”,“mg/dL”,2020-09-08 20:28:11 -0400,2020-09-08 20:28:00 -0400,2020-09-08 20:28:00 -0400,97
“Dexcom G6”,“16018”,“BloodGlucose”,“mg/dL”,2020-09-08 20:28:12 -0400,2020-09-08 17:28:00 -0400,2020-09-08 17:28:00 -0400,103
No heatmap is generated.
No heatmap, or an empty one? The default correlation filter is .5, so that might generate an empty one. Otherwise, it might be a data issue I haven’t expected. Does the CSV have data? Do you notice any unusual formatting issues with either columns or data?
Sorry, an empty one. Yep, the CSV has tons of BG data…
Yep, you are right, it is probably the correlation filter!
I think I’m going to have to merge the csv files. For example, BloodGlucose.csv only has one variable, value (=BG), so I’m guessing this is why the heatmap is empty!
Luckily, I’ve avoided that issue. The Google Fit data, while broken out into many separate files, also has a summary file, which is what I’ve been using.
That’s great. Maybe I should take a look at the summary file to see what format Apple Health should produce. At least I have the data so it definitely can be parsed maybe with some extra data manipulation!
Yes, load the files into data frames then use pandas’ concat. It depends on your data, but…
I did more work, improved the structure of the workbook, added some generic functions, and then charted some of the worthwhile relationships. Also, after I moved my personal data to a private repo, I made the repo public. You might want to look it over and see if any improvements could be useful, and please let me know if you see something missing that I should be applying to my data.
Thanks for the update @JamesIgoe. Looks like great progress. I like the additional graphs, as well as the heatmap. My Apple Health data does not look quite so refined as your Google Fit data since each of my categories are basically in a separate csv, but with some manipulation I could probably merge them to generate summary data like your Google Fit.
One oddity that I ran across, sleep wasn’t the entirety of my sleep. It accounted for 1/10 of the actual number, but after I combined that with deep, light, and awake mid-sleep, it added up.
Thanks for the heads up on the sleep oddity. I’ll check it out in the Apple Health data.
One thing I did today was to shift my fitness data by a day. I was getting some correlations between my rowing workouts, sleep, and systolic blood pressure, but I wanted to see the effect of fitness on sleep and BP, but since it occurred the day before sleep and blood pressure, needed to shift the fitness data forward one day (using .shift() in Python, into a new column, divided by 60000 to convert milliseconds to minutes ) so they all occurred on the same day. The BP relationship disappeared, but total sleep was still strongly positively correlated.
Interesting. Thanks for the tip. This makes sense.
Most insurance companies I’ve dealt with will cover cartridges for fast acting insulin… basal, probably not, it tends to be cartridges or disposable pens, just have to find out which type is covered and tell the dr you need cartridges… I don’t recommend the in pen…I have one but I could not integrate it with xdrip, their app did not integrate Cgm data, so I had to log doses on xdrip anyway, the cartridge changes were the worst imaginable, and towards the end of the cartridge, it will not let you know how much insulin is left…horrible… I personally use the Owen Mumford autopen and the humapen savvio(had to buy from Canada because its not fda approved), both have much better delivery mechanisms, and no useless app sending notifications all day about skipping meals, etc
Yeah, I am one of the very fortunate ones! I am usually in the usual patterns for any system I have used. And when I first started on the Dexcom G6, one of the endocrinologist I work with on some clinical trials (who is also type 1j, suggested to just let go. And at first I was in panic mode, but I did let go and just let the system do it’s thing. Been using it with the Tandem pump and have had no problems. I have only pulled out my meter once in a great while if I am in warmup period and want to eat. Otherwise, no testing. Been a very cool aspect to this new system for me. No more thinking all the time about my diabetes. I never worry about overnights. And if I run low and can’t get to tablets right away, I am very sensitive to the no insulin time. It doesn’t take long for things to come back into line. On my Clarity reports, no very highs. High % is now always single digit versus major double digits. Low under 5% and very low is gone. So incredibly sweet how easy it is.