Lesson 2.3

Terms of Service

Learning Objectives

After this, students can explain why a platform’s terms of service (ToS) matter legally, especially once you create an account.
After this, students can find and read the ToS sections that matter for OSINT: acceptable use, automated access, account rules, and API terms.
After this, students can use robots.txt, official APIs, and research access programs to choose approved collection methods.
After this, students can read a ToS “backwards” to spot what a platform can technically do, while keeping possible and permitted separate.
After this, students can summarize hiQ v. LinkedIn, Meta v. Bright Data, and X Corp v. Bright Data and what each means for scraping.

Why the ToS Matters

A platform’s terms of service are a contract between the platform and its users. Violating them is usually not a crime by itself (see 2.2-computer-crime-and-privacy-laws.md for the 2022 DOJ policy and Sandvig v. Barr), but it can still lead to:

Account bans, which can cut you off from a source you depend on for investigations
Civil lawsuits for breach of contract and related claims
Evidence problems, if a client, court, or editor questions how information was obtained
Reputational damage for you and your employer

When are you bound by the ToS?

Courts look at whether you actually agreed to the terms:

Clickwrap: you click “I agree” (for example, when creating an account). These are generally enforceable.
Browsewrap: the terms are just linked somewhere on the page and say that using the site means you agree. These are much harder to enforce, because a visitor may never have seen them.

This distinction explains why logged-in vs. logged-out collection matters so much in scraping cases (see the case studies below). Once you create an account, you have almost certainly agreed to the platform’s rules.

Read the ToS Before You Start

Before collecting from any platform, spend ten minutes with its legal pages. Look for these sections:

Section (names vary)	What to look for	Why it matters for OSINT
Acceptable use / “Dos and Don’ts”	Prohibited behaviors	The platform’s list of what you may not do
Automated access / scraping	Rules on bots, crawlers, scripts, browser extensions	Determines whether tools like scrapers are allowed, and under what conditions
Account requirements	Real-name rules, one account per person, age limits	Directly affects sock puppet accounts
API / developer terms	Rate limits, allowed use cases, data retention, redistribution	Often the approved path for bulk collection
Privacy policy	What data the platform collects and shares	Tells you what data exists (see “Reading the ToS for hints” below)
Research or data access programs	Special access for researchers or journalists	A legitimate alternative to scraping
Termination	What happens if you break the rules	Bans, loss of access, legal action
Governing law / disputes	Which courts or arbitration apply	Where you could be sued

Real examples

Meta (Facebook) Terms of Service include rules that directly affect OSINT work:
- “Provide for your account the same name that you use in everyday life.”
- “Create only one account (your own) and use it for personal purposes.”
- “You may not access or collect data from our Products using automated means (without our prior permission)”

LinkedIn User Agreement (“Dos and Don’ts”) says users may not:
- “Create a false identity on LinkedIn, misrepresent your identity, create a Member profile for anyone other than yourself (a real person), or use or attempt to use another’s account”
- “Develop, support or use software, devices, scripts, robots or any other means or processes (such as crawlers, browser plugins and add-ons or any other technology) to scrape or copy the Services, including profiles and other data”
- It also prohibits using bots or other unauthorized automated methods to access the services, add or download contacts, or send messages, and prohibits sharing login credentials or copying cookies to access others’ accounts.

(Terms change often. Quote the current version when you teach this.)

robots.txt, APIs, and Approved Access

robots.txt

robots.txt is a plain text file at the root of a website (for example, https://example.com/robots.txt) that tells automated crawlers which paths they should and shouldn’t access. It was a de facto convention from 1994 until it was formalized as RFC 9309 (Robots Exclusion Protocol) in 2022.

User-agent: *
Disallow: /admin/
Disallow: /internal-search/
Crawl-delay: 10

User-agent: ExampleResearchBot
Allow: /

Key points:
- It expresses the site owner’s wishes for automated clients. It is not a security control, and RFC 9309 itself notes that listing paths makes them publicly discoverable.
- Ignoring it isn’t automatically illegal, but it’s evidence that you knew the owner didn’t want automated access, which can matter in a dispute.
- Crawl-delay isn’t part of the RFC 9309 standard, but some crawlers honor it as a hint about acceptable request rates.
- For OSINT, robots.txt does double duty: it tells you what the owner doesn’t want crawled, and it can reveal interesting paths (see Module 5). Viewing an archived copy keeps you passive.

Official APIs

Many platforms offer an API (application programming interface) as the approved way to access data programmatically. APIs come with:
- Documented rate limits and allowed use cases
- Terms about storing, sharing, and deleting data
- Often a cost, and sometimes an approval process

If an API exists for what you need, using it is almost always safer than scraping the website.

Research and data access programs

Some platforms run programs that give vetted researchers or journalists access to data, and some laws are pushing platforms in that direction (for example, the EU’s Digital Services Act includes provisions for vetted researcher access to platform data (verify current details before class)). If you’re doing academic or public-interest research, check for these programs before building your own collection method.

Other “rules files”

security.txt (RFC 9116): a file at /.well-known/security.txt telling researchers how to report vulnerabilities. If you stumble on an exposure during OSINT, this is where to look for the right contact (see 2.4-ethics-and-harm.md).
Bug bounty / vulnerability disclosure policies: define exactly what testing is permitted.

Reading the ToS for Hints: What’s Possible

Rules are usually written because something can be done, and often because someone already did it. That makes the ToS, privacy policy, and developer documentation an overlooked source of intelligence about how a platform works.

If the terms say…	It suggests…	How an investigator or defender uses that lead
You may not collect data “using automated means”	Data on the platform can be collected at scale	Expect that your organization’s public profiles have already been scraped by someone; plan defenses accordingly
You may not “add or download contacts” using bots	Contact lists can be bulk exported	Contact exports are a real exposure path worth covering in awareness training
You may not copy cookies to access others’ accounts	Session cookies can grant account access	Stolen session cookies (e.g., from infostealer malware, Module 6) are a real threat; defenders should plan for session revocation
You may not use the API beyond what your app needs	The API may return more fields than the website shows	Read the API documentation to understand what data exists
You may not use the service to track someone’s location	Location data exists somewhere in the platform	Check what location features exist and what users expose, then advise people on privacy settings
The privacy policy lists “precise location,” “contacts,” “device identifiers”	The platform stores those data types	Use the platform’s “download your data” tool on your own account to see exactly what’s stored

Developer and API documentation works the same way: it lists the fields, endpoints, and features that exist, which helps you understand what a platform knows even when the website doesn’t show it.

Treat these as leads, not proof. A clause might be boilerplate, might describe a feature that has been removed, or might be written defensively for something that was never possible.

Possible is not permitted. Knowing that data can be collected doesn’t mean you’re allowed to collect it. A prohibition in the ToS is still a prohibition, and it stays off-limits unless you get permission or use an approved channel.

Sock Puppets and the ToS

A sock puppet is a fictional research account used to view content without exposing your real identity (see Module 8 for how investigators manage them). The ToS problem is direct:

Meta requires the name you use in everyday life and only one account.
LinkedIn prohibits creating a false identity.
In hiQ v. LinkedIn, using crowdsourced workers to create fake profiles was one of the two reasons the court found hiQ breached LinkedIn’s User Agreement.

At the same time:
- The 2022 DOJ policy lists “using a pseudonym on a social networking site that prohibits them” and “creating fictional accounts on hiring, housing, or rental websites” as conduct that, by itself, isn’t enough for federal CFAA charges.
- Sandvig v. Barr (2020) concluded that the CFAA doesn’t criminalize mere ToS violations on consumer websites.

What this means in practice:
- Sock puppets are usually a contract and platform-policy risk rather than a criminal one, but that risk is real (bans, lawsuits, questions about evidence).
- Professional teams decide whether to use them with legal guidance, and document the decision.
- Never impersonate a real person or use a real person’s photos. Impersonating an actual person is a different and more serious problem, and some states criminalize certain forms of online impersonation (verify specific state laws before class).
- For this class, sock puppets are not allowed unless the instructor sets up a specific, approved exercise.

Case Studies: Scraping and the ToS

hiQ Labs v. LinkedIn (2017–2022)

hiQ scraped public LinkedIn profiles. Courts suggested this was likely not a CFAA violation.
In November 2022, the district court found hiQ breached LinkedIn’s User Agreement through automated scraping and fake profiles.
In December 2022, hiQ agreed to a consent judgment: a permanent injunction, deletion of its scraped data and code, and $500,000 in damages.
Lesson: Account-based collection and fake accounts made the ToS enforceable against hiQ.

Meta Platforms v. Bright Data (2023–2024)

Bright Data sold data scraped from Facebook and Instagram while logged off (no Meta account signed in).
In January 2024, Judge Edward Chen granted summary judgment to Bright Data, finding that Meta’s terms didn’t apply to and prohibit Bright Data’s scraping of publicly available data while logged off. A factor in the ruling: Meta had revised its terms in 2009 to remove language saying that anyone who visited Facebook, user or not, was bound by them.
On February 26, 2024, Meta dropped its remaining claim and waived its right to appeal.
Lesson: Terms bind users. Logged-off collection of public data was treated differently from logged-in collection.

X Corp. v. Bright Data (2023–2024)

X sued Bright Data for scraping and selling publicly available data from X, claiming it violated X’s terms and policies.
On May 9, 2024, Judge William Alsup dismissed X’s claims, holding that federal copyright law preempted X’s contract-based claims. Enforcing them would let X control public content that it doesn’t own the copyright to, and block fair uses the Copyright Act allows.
The court noted that claims based on a substantial non-copyright interest, such as protecting user privacy, might not be preempted.
(Check the case’s later status before class.)
Lesson: Courts are still working out how far a platform’s ToS can reach over public data, and the answer depends on the facts and the claims.

Summary: logged in vs. logged out

Situation	ToS risk
Viewing public pages while logged out	Low; you likely never agreed to the terms
Scraping public pages while logged out	Contested; recent cases favored the scraper, but other claims can apply
Viewing or collecting while logged in to your real account	You agreed to the terms, so their rules apply to you
Collecting through fake accounts	High; breach of contract, and it undermined hiQ’s position
Using the official API within its terms	Low; this is the approved path

A Pre-Collection Checklist

Before collecting from a platform, answer these questions:

Have I read the current ToS, privacy policy, and any developer terms?
Do I need an account? If so, what have I agreed to?
Does the ToS allow automated collection? If not, is there an API or research program?
What does robots.txt say about the paths I want?
Does my engagement scope (see 2.1-authorization-and-scope.md) allow this source and method?
Am I planning to use a sock puppet, and has that been approved?
If the ToS says no: can I use a different source, a different method, or get explicit permission?

Key Terms

Terms of Service (ToS): The contract governing use of a platform. Also called terms of use or a user agreement.
Clickwrap: Terms you actively accept, such as by clicking “I agree.” Generally enforceable.
Browsewrap: Terms that claim to apply just because you use a site. Harder to enforce.
Scraping: Automated collection of data from websites.
API (application programming interface): A documented, approved way for programs to access a platform’s data.
robots.txt: A file telling automated crawlers which paths to avoid, standardized in RFC 9309.
security.txt: A file at /.well-known/security.txt listing how to report security issues, standardized in RFC 9116.
Sock puppet: A fictional account used for research without revealing the investigator’s identity.
Preemption: A legal doctrine where federal law overrides conflicting state law claims.

Key Takeaways

Read the ToS before you collect: it’s the platform telling you what you can and can’t do.
Creating an account usually means you’ve agreed to the platform’s rules.
Use official APIs and research programs when they exist; they’re the approved path.
robots.txt shows what the owner doesn’t want crawled; it isn’t security, but ignoring it can hurt you in a dispute.
Prohibitions hint at what’s technically possible. Possible is not permitted.
Fake accounts helped turn hiQ v. LinkedIn into a clear breach of contract.
Meta v. Bright Data: logged-off scraping of public data wasn’t covered by Meta’s terms.
If the ToS says no, change your method or source, or get permission.

References

Free to use, copy, change and teach. Security For the Folks dedicates this course to the public domain under CC0 1.0 Universal: no permission or credit needed. Quoted material and the sources it links to belong to their own authors.