Methodology — Are We Screwed?

Transparency notice

This database uses AI-assisted research with human review. We use Claude (by Anthropic) to help research products, identify contradictions, and draft findings. Every finding is reviewed and edited by a human researcher before publication. We believe this is the only way to investigate at scale — 684 products and counting — while maintaining quality. This page explains exactly how the process works so you can judge the output for yourself.

Source health

We cite 3,443 sources across 684 products. Sources include FTC filings, court documents, news articles, academic papers, CVE records, and government reports. We periodically verify these URLs — currently 73.2% verified as live (2,520 of 3,443). Sources that have moved or been deleted are shown with a strikethrough and marked "(link unavailable)" on device pages. The source title is preserved so you can search for the original content independently.

News sites restructure URLs, government agencies redesign their websites, and publications shut down (Vice, BuzzFeed News). When a URL breaks, we look for the new location. When no replacement exists, we keep the citation — the finding still happened, even if the link moved. If you find a broken source and know the new URL, let us know.

When a source link is unavailable

Where source links are unavailable, the underlying findings are derived from information present in publicly available sources at the time of research. Our AI-assisted research draws on a large language model whose training data reflects publicly available internet content up to its knowledge cutoff. This means the facts behind each finding were documented in public sources — news articles, court filings, regulatory decisions, academic papers — but the specific URL we cited may have since moved, been restructured, or taken offline.

What this means in practice: A finding with an unavailable link is not fabricated, but it may not be independently verifiable without further searching. The source title is preserved alongside every finding, so you can search for the original content by title. We are actively working to restore or replace broken links with verified URLs.

What we cannot guarantee: AI-assisted research can occasionally merge, misattribute, or subtly misrepresent details from multiple real sources. Severity ratings and plain-language summaries are editorial interpretations, not direct quotations. We encourage readers to verify any finding that informs a consequential decision.

What we investigate

We examine products across seven evidence layers. A contradiction is found when what a company says in one layer doesn't match what's observed in another. Different product types use different layers:

Layer	What it captures
Policy claims	Privacy policy, EULA, terms of service
Marketing claims	Website copy, advertising, PR statements
App permissions	Companion app access requests and embedded trackers
Network analysis	DNS queries, packet captures, observed data transmission
Firmware analysis	Chipsets, FCC filings, teardowns, hidden hardware capabilities
Regulatory findings	FTC complaints, court filings, ACCC actions, GDPR decisions
Third-party research	Independent audits, academic papers, journalist investigations

How findings are produced

Product selection. We prioritise products people actually use, products with known controversies, and products where public perception doesn't match evidence (e.g. Apple “privacy-first”).
Evidence gathering. We collect publicly available evidence: privacy policies, app store listings, Exodus Privacy tracker reports, FCC filings, court documents, academic studies, news investigations, and (where possible) network traffic captures from physical devices.
AI-assisted analysis. Claude identifies potential contradictions between evidence layers. This is where AI provides scale — it can cross-reference a privacy policy against 50 news articles faster than a human researcher.
Human review. Every contradiction is reviewed by a human. We check: Is the claim accurate? Is the evidence real and correctly attributed? Does the plain-language summary fairly represent both sides? Is the severity rating justified?
Source citation. We are actively adding verifiable source links to every contradiction. Each source should allow a reader to click through and verify the finding independently within 30 seconds.
Publication. Findings are stored in a structured database, then rendered into the static site you're reading. The site itself uses zero external dependencies, no tracking, no analytics, no cookies.

Source citation progress

4,359

Total findings

2,845

With source links

3,443

Total sources

73.2%

URLs verified live

We are working to add direct source URLs to every finding. On each device page, findings with sources show clickable links in the expanded details. Our goal is 100% source coverage before public launch.

Severity ratings

Each contradiction is rated by severity based on real-world impact:

Critical Active harm to users. Data breaches with exposed records, law enforcement access to private data, documented physical or psychological harm.

High Significant deception or risk. Company claims directly contradicted by evidence, covert data collection, misleading marketing.

Medium Concerning practices. Excessive data collection, opaque policies, dark patterns in privacy settings.

Low Minor issues. Industry-standard practices that could be better, minor inconsistencies between documents.

Grading

Products are graded A+ through F based on average contradiction severity. Software products use stricter thresholds than hardware because software has more publicly available research (more evidence means more contradictions for even average products). The grade reflects what we found, not an overall product quality judgement. A product with fewer publicly documented issues may simply have less available evidence.

What we get wrong

We make mistakes. AI-assisted research at scale means some findings may be imprecise, out of date, or missing context. Common failure modes:

Stale information. A company may have changed practices since the evidence was gathered. We try to date our sources but some findings may not reflect the current state.
Missing context. A finding may be technically accurate but miss an important nuance — for example, a feature that was briefly available in one region but not others.
Severity disputes. Reasonable people can disagree about whether something is “critical” or “high.” Our ratings reflect our assessment of real-world harm.
Attribution. Some practices are industry-wide but we attribute them to specific products. We try to note when something is standard practice vs uniquely bad.

Found an error?

If any finding is inaccurate, misleading, or missing important context, we want to know. We take corrections seriously because the credibility of every finding depends on getting the details right. Email us with the product name, the specific finding, and what's wrong. We will review and correct or remove any finding that doesn't meet our evidence standard.

Click to reveal email address

Independence

DeviceGuardian is an independent research project. We have no commercial relationships with any product we investigate. We don't accept payment for reviews, and we don't modify findings based on company requests. Our REPLACE recommendations (suggesting alternatives to poorly-graded products) are based on evidence, not partnerships. The site itself collects zero user data — no analytics, no cookies, no tracking pixels. We practice what we preach.