Skip to content

[MRG] adding support for apple health #44

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
gedankenstuecke merged 4 commits into from
May 16, 2020
Merged

[MRG] adding support for apple health #44

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
5179616
add support for apple health
gedankenstuecke May 14, 2020
03c29c7
add comment
gedankenstuecke May 15, 2020
f08547d
(minor) improve template code formatting
madprime May 16, 2020
bc81a34
Merge pull request #2 from madprime/adding-apple-health
gedankenstuecke May 16, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 31 additions & 4 deletions import_data/activity_parsers.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,31 @@ def fitbit_parser(fitbit_info, event_start, event_end=None):

return returned_fitbit_data


def apple_health_parser(apple_health_info, event_start, event_end=None):
if not event_end:
event_end = event_start
apple_health_data = requests.get(apple_health_info["download_url"]).text
apple_health_data = apple_health_data.split("\n")
start_date = arrow.get(event_start)
end_date = arrow.get(event_end)
period_start = start_date.shift(weeks=WEEKS_BEFORE_SICK * -1)
period_end = end_date.shift(weeks=WEEKS_AFTER_SICK)

returned_apple_data = []

for entry in apple_health_data:
if entry.endswith("R"):
entry = entry.split(",")
sdate = arrow.get(entry[1])
if sdate >= period_start and sdate <= period_end:
returned_apple_data.append(
{"timestamp": entry[1], "data": {"heart_rate": entry[0]},}
)

return returned_apple_data


def googlefit_to_qf(json_data, min_date, max_date):
res = []
data = json_data
Expand All @@ -96,7 +121,7 @@ def googlefit_to_qf(json_data, min_date, max_date):
for col_name in df.columns:
# the col_name should be of the format heart_rate.bpm.INT
# named this way by the parsing functionality in get_dataframe_with_all
if 'heart_rate' not in col_name:
if "heart_rate" not in col_name:
continue
data_points = len(df[col_name].dropna())
if data_points > max_data_points:
Expand All @@ -112,7 +137,7 @@ def googlefit_to_qf(json_data, min_date, max_date):
ts = ts.to_pydatetime()
if ts < min_date or ts > max_date:
continue
rec = {"timestamp": ts.isoformat(), "data":{"heart_rate": value}}
rec = {"timestamp": ts.isoformat(), "data": {"heart_rate": value}}
res.append(rec)
return res

Expand All @@ -121,7 +146,9 @@ def googlefit_parser(googlefit_files_info, event_start, event_end=None):
print(event_start)
if event_end is None:
event_end = event_start
event_start = datetime(event_start.year, event_start.month, event_start.day, 0, 0, 0)
event_start = datetime(
event_start.year, event_start.month, event_start.day, 0, 0, 0
)
event_end = datetime(event_end.year, event_end.month, event_end.day, 23, 59, 59)
min_date = event_start - timedelta(days=21)
max_date = event_end + timedelta(days=14)
Expand All @@ -143,7 +170,7 @@ def googlefit_parser(googlefit_files_info, event_start, event_end=None):

data_in_qf_format = googlefit_to_qf(googlefit_json, min_date, max_date)
if data_in_qf_format:
returned_googlefit_data+=data_in_qf_format
returned_googlefit_data += data_in_qf_format

return returned_googlefit_data

Expand Down
13 changes: 11 additions & 2 deletions quantified_flu/helpers.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,16 +9,25 @@


def identify_missing_sources(oh_member):
missing_sources = {"oura": True, "fitbit": True, "googlefit": True}
missing_sources = {
"oura": True,
"fitbit": True,
"googlefit": True,
"apple_health": True,
}

# Check data already in Open Humans.
for i in oh_member.list_files():
if i["source"] == "direct-sharing-184" and i["basename"] == "oura-data.json":
missing_sources.pop("oura", None)
if i["basename"] == "fitbit-data.json" and i["source"] == "direct-sharing-102":
missing_sources.pop("fitbit", None)
if "googlefit" in i["basename"]: # GoogleFit is stored in monthly files
if "googlefit" in i["basename"]: # GoogleFit is stored in monthly files
missing_sources.pop("googlefit", None)
if i["source"] == "direct-sharing-453" and i["basename"].startswith(
"heartrate_samples"
):
missing_sources.pop("apple_health", None)

@carolinux carolinux May 15, 2020

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alright so this PR handles the case of the data having been imported by OH already, and doesn't offer an option to import them through QF. Is this planned for future?

@gedankenstuecke gedankenstuecke May 15, 2020

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nope, the Apple Health data is a bit special, as you can't export it through an API or anything, but just through having the corresponding iPhone app installed that exports data. Which makes integrating it much harder.

To integrate this in QF we'd need to generate a whole new API to which to export the data (not to mention the fact that it took 2 months to get the app into the appstore :D).

So using the app we got now seems the way to go (plus it doesn't make much difference, as people need to use an external tool in any case).

@carolinux carolinux May 15, 2020

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see :D

@madprime madprime May 16, 2020

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this bit of code reminds me that supporting the google fit data from the separate activity is missing :) I created #46

@gedankenstuecke gedankenstuecke May 16, 2020

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @madprime!


# Check data imported by this app.
if hasattr(oh_member, "fitbit_member"):
Expand Down
1 change: 1 addition & 0 deletions quantified_flu/templates/quantified_flu/about.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,7 @@ <h2 class='panel-title'>Which wearables are supported?</h2>
Right now we support the following wearable devices:
<ul>
<li>All wearables by <i>Fitbit</i></li>
<li><i>Apple Watch</i> (and all other devices that deposit in <i>Apple Health</i>)</li>
<li>The <i>Oura</i> Ring</li>
<li>All wearables that can export to <i>Google Fit</i>, e.g. Android Smart Watches.</li>
</ul>
Expand Down
62 changes: 47 additions & 15 deletions quantified_flu/templates/quantified_flu/home.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ <h4 class="text-center text-muted">Whether the flu or the common cold: Can weara
<p>We have two ways you can contribute…</p>
<h3>Report past illness</h3>
<p>
Report when you got sick and share your wearable device data. <b>We'll plot your data for you!</b> Currently we support <em>Fitbit</em> & <em>Oura Ring</em> and <em>Google Fit</em>.
Report when you got sick and share your wearable device data. <b>We'll plot your data for you!</b> Currently we support <em>Fitbit</em> & <em>Oura Ring</em>, <em> Apple Watch</em> and <em>Google Fit</em>.
<a href='https://docs.google.com/forms/d/e/1FAIpQLSdDjzLyL_HJ79ij4GbepqeNYAyYf4LWapXFyIDFwMRdXOBBXg/viewform'>Ask us to add support for your wearable</a>, if yours is missing so far!
</p>
<h3>Track symptoms going forward</h3>
Expand All @@ -37,7 +37,7 @@ <h3>Using your data for good</h3>

{% if request.user.is_authenticated %}

{% if missing_sources|length == 3 %}
{% if missing_sources|length == 4 %}
<h2>Data Sources</h2>
<p>
You can connect these data sources to correlate them with reported illness.
Expand All @@ -53,10 +53,16 @@ <h2>Data Sources</h2>
Connect Oura account
</a>
</div>
<a
href="{%url 'import_data:authorize-googlefit' %}" class="btn btn-primary btn-lg">
Connect GoogleFit account
</a>
<div class="col-md">
<a
href="https://apps.apple.com/us/app/oh-data-port/id1512384252" class="btn btn-primary btn-lg">
Connect Apple Watch
</a>
</div>
<a
href="{%url 'import_data:authorize-googlefit' %}" class="btn btn-primary btn-lg">
Connect GoogleFit account
</a>
</div>
<hr>
{% endif %}
Expand Down Expand Up @@ -102,7 +108,7 @@ <h4>Schedule check-ins</h4>
If at least one data source has been connected, management of data sources is
moved to below the "actions" (retrospective analysis & scheduling check-ins).
{% endcomment %}
{% if missing_sources|length < 3 %}
{% if missing_sources|length < 4 %}
<hr>
<h2>Data Sources</h2>
{% comment %}
Expand All @@ -129,6 +135,11 @@ <h2>Data Sources</h2>
<li>Connect GoogleFit account</li>
</a>
{% endif %}
{% if source == 'apple_health' %}
<a href="https://apps.apple.com/us/app/oh-data-port/id1512384252">
<li>Connect Apple Watch / Apple Health</li>
</a>
{% endif %}
{% endfor %}
</ul>
</p>
Expand Down Expand Up @@ -169,9 +180,10 @@ <h3>Manage Fitbit</h3>
You can manage this connection at:<br>
<a href="https://fitbit.openhumans.org">fitbit.openhumans.org</a>
</p>
{% endif %}
{% endif %} {# if request.user.openhumansmember.fitbit_member #}
</div>
{% endif %}
{% endif %} {# if 'fitbit' not in missing_sources #}

{% if 'oura' not in missing_sources %}
<div class="col-md-6">
<h3>Manage Oura</h3>
Expand Down Expand Up @@ -202,10 +214,30 @@ <h3>Manage Oura</h3>
You can manage this connection at:<br>
<a href="https://oura.openhumans.org">oura.openhumans.org</a>
</p>
{% endif %}
{% endif %}
{% endif %} {# if request.user.openhumansmember.oura_user #}
</div>
{% endif %} {# if 'oura' not in missing_sources #}

{% if 'googlefit' not in missing_sources %}
{% if 'apple_health' not in missing_sources %}
<div class="col-md-6">
<h3>Manage Apple Watch / Apple Health </h3>

<p>
<b>Your data files:</b> You've shared Oura data via the
Open Humans importer app. Your data file (<code>"heartrate_samples*.csv"</code>)
should be available in your Open Humans account here:<br>
<a href="https://www.openhumans.org/activity/oh-data-port-for-apple-health/#activity-panel-data">
https://www.openhumans.org/activity/oh-data-port-for-apple-health/
</a>
</p>
<p>
You can update your data through the
<a href="https://apps.apple.com/us/app/oh-data-port/id1512384252">iPhone app</a>.
</p>
</div>
{% endif %} {# if 'apple_health' not in missing_sources #}

{% if 'googlefit' not in missing_sources %}
<div class="col-md-6">
<h3>Manage GoogleFit</h3>
{% if request.user.openhumansmember.googlefit_member %}
Expand Down Expand Up @@ -235,11 +267,11 @@ <h3>Manage GoogleFit</h3>
You can manage this connection at:<br>
<a href="https://google-fit.openhumans.org">google-fit.openhumans.org</a>
</p>
{% endif %}
{% endif %} {# if request.user.openhumansmember.googlefit_member #}
</div>
{% endif %}
{% endif %} {# if 'googlefit' not in missing_sources #}
</div> <!-- .row -->
{% endif %} {# if missing_sources < 3 #}
{% endif %} {# if missing_sources < 4 #}
{% endif %} {# if user.is_authenticated #}

{% endblock main %}
66 changes: 53 additions & 13 deletions retrospective/tasks.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,8 @@
oura_parser,
fitbit_parser,
fitbit_intraday_parser,
googlefit_parser
googlefit_parser,
apple_health_parser,
)
from import_data.celery_fitbit import fetch_fitbit_data
from import_data.celery_oura import fetch_oura_data
Expand Down Expand Up @@ -45,6 +46,7 @@ def get_wearable_data(member_files):
fitbit_data = None
googlefit_data = []
fitbit_intraday_data = []
apple_health_data = None

# Get info for relevant files.
# Take the first match for Fitbit and Oura, where all data is in a single file.
Expand Down Expand Up @@ -76,11 +78,15 @@ def get_wearable_data(member_files):
if source == "direct-sharing-191":
fitbit_intraday_data.append(file_info)

if source == "direct-sharing-453" and basename.startswith("heartrate_samples"):
apple_health_data = file_info

return {
"oura_data": oura_data,
"googlefit_data": googlefit_data,
"fitbit_data": fitbit_data,
"fitbit_intraday_data": fitbit_intraday_data,
"apple_health_data": apple_health_data,
}


Expand All @@ -90,12 +96,13 @@ def analyze_googlefit_event(googlefit_data, event):
googlefit_hr_analysis = RetrospectiveEventAnalysis(
event=event,
graph_data=json.dumps(googlefit_hr_data),
graph_type="googlefit_heartrate"
graph_type="googlefit_heartrate",
)
googlefit_hr_analysis.save()
else:
print("No GoogleFit data available around event {}".format(event.date))


@task
def analyze_event(event_id):
event = RetrospectiveEvent.objects.get(id=event_id)
Expand All @@ -106,6 +113,7 @@ def analyze_event(event_id):
fitbit_data = wearable_data["fitbit_data"]
fitbit_intraday_data = wearable_data["fitbit_intraday_data"]
googlefit_data = wearable_data["googlefit_data"]
apple_health_data = wearable_data["apple_health_data"]

if googlefit_data:
analyze_googlefit_event(googlefit_data, event)
Expand All @@ -130,6 +138,20 @@ def analyze_event(event_id):
)
oura_temp_analysis.save()

if apple_health_data:
apple_analyses = event.retrospectiveeventanalysis_set.filter(
graph_type__exact="apple_health_summary"
)
if not apple_analyses:
apple_hr_data = apple_health_parser(apple_health_data, event.date)
if apple_hr_data:
apple_hr_analysis = RetrospectiveEventAnalysis(
event=event,
graph_data=json.dumps(apple_hr_data),
graph_type="apple_health_summary",
)
apple_hr_analysis.save()

if fitbit_data:
fitbit_analyses = event.retrospectiveeventanalysis_set.filter(
graph_type__exact="fitbit_summary"
Expand All @@ -153,15 +175,20 @@ def analyze_event(event_id):
)

if not fitbit_intraday_analyses:
fb_intraday_data = fitbit_intraday_parser(
fitbit_data, fitbit_intraday_data, event.date
)
new_analysis = RetrospectiveEventAnalysis(
event=event,
graph_data=json.dumps(fb_intraday_data),
graph_type="fitbit_intraday",
)
new_analysis.save()
try:
fb_intraday_data = fitbit_intraday_parser(
fitbit_data, fitbit_intraday_data, event.date
)
new_analysis = RetrospectiveEventAnalysis(
event=event,
graph_data=json.dumps(fb_intraday_data),
graph_type="fitbit_intraday",
)
new_analysis.save()
except:
# The fitbit intraday parsing is very fickle and often breaks
# Blanket except here to allow the pipeline to work regardless.
pass

@carolinux carolinux May 15, 2020

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in which case would an exception be thrown here? maybe add a comment

@gedankenstuecke gedankenstuecke May 15, 2020

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, it's a rather general catch-all as the merging of Fitbit & Fitbit Intraday is really fickle and easily crashes (e.g. I stopped using my Fitbit some time ago but still have my data in my account. Fitbit still updates the data even and reports 0 for each day, this leads the whole pipeline to crash 🙈 ).

@madprime madprime May 16, 2020

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not ideal but better than a server error ruining other processing/response? wouldn't hold this up based on improving this further



@task
Expand Down Expand Up @@ -192,7 +219,6 @@ def update_googlefit_data(oh_id, django_user_id):
analyze_existing_reports(django_user_id)



def set_symptomwearablereport(oh_member, data_source, start, end, data):
values = json.dumps(data)
wearable_report, created = SymptomReportPhysiology.objects.get_or_create(
Expand All @@ -216,6 +242,7 @@ def add_wearable_to_symptom(oh_member_id):
fitbit_data = wearable_data["fitbit_data"]
fitbit_intraday_data = wearable_data["fitbit_intraday_data"]
googlefit_data = wearable_data["googlefit_data"]
apple_health_data = wearable_data["apple_health_data"]

symptoms_start = (
SymptomReport.objects.filter(member=oh_member).earliest("created").created
Expand All @@ -233,7 +260,20 @@ def add_wearable_to_symptom(oh_member_id):
data_source="googlefit_heartrate",
start=symptoms_start,
end=symptoms_end,
data=googlefit_hr_data)
data=googlefit_hr_data,
)

if apple_health_data:
apple_hr_data = apple_health_parser(
apple_health_data, symptoms_start, symptoms_end
)
set_symptomwearablereport(
oh_member=oh_member,
data_source="apple_health_summary",
start=symptoms_start,
end=symptoms_end,
data=apple_hr_data,
)

if oura_data:
oura_hr_data, oura_temp_data = oura_parser(
Expand Down
Loading