How to scrape social media for AI research

Social posts can tell you what product pages leave out: what broke, what people tried, and what they paid. To research those reports with AI, first collect the posts and replies into a dataset you can inspect. Then ask the model to organize the evidence and help you check it.

This guide uses scraping to mean collecting that content into local files. The practical route depends on the platform: an official API, an authorized export, or another collection method it permits. The steps are plan, collect, analyze and check. A downloadable Hacker News collector below shows the collection step without an account or API key.

You do not always need a separate scraper. Some AI tools can retrieve social content through licensed access, including Reddit’s partnership with OpenAI. Other tools have restrictions: users have reported Claude blocking Reddit. Check whether your tool can retrieve the actual posts and replies before relying on its summary.

Step 1: Choose the question and the sample

Start with a question you can answer from experience reports. For example: “What problems do trainers report when using AI avatar videos?” Choose the communities, search terms and date range before collecting. Record those choices so someone else can understand how you selected the material.

Run the searches and save the URLs or post IDs you intend to collect. Include reports that disagree with your initial expectation. Keep a search log with the query, platform, search date and selection rule. Convenient search results can help you explore a question, but they are not a representative sample of users.

Step 2: Collect posts and replies

Use the platform’s supported route to collect the post body, replies and source metadata. Check access requirements before writing the script. Being able to view a page does not, by itself, establish permission to copy its contents or upload them to another service.

• Hacker News: find discussions with Algolia search, then collect each item and its replies through the official API. It needs no key. Each item’s kids field lists its direct replies; follow those IDs recursively.
• Reddit: obtain approved API access for your use case, then collect selected submissions and their comment trees. The Data API terms require a separate agreement for commercial use and some other uses. The user agreement also restricts automated and nonautomated collection. Copying by hand is not a blanket exception.
• YouTube: collect discussion with commentThreads.list. Embedded replies can be incomplete; use comments.list with the parent comment ID to retrieve the replies. Follow nextPageToken until it is absent.
• Bluesky: use post search to find candidates, then retrieve the threads you selected. Host-dependent search failures have been reported; check current authentication requirements and record failures. A failed request does not mean no posts exist.
• Mastodon: use the chosen server’s API. Full-text post search requires authentication and a server with the search backend enabled. State which server you queried; its results are not a complete search of every server.
• X: use the official API and set a collection budget. As checked on October 14, 2026, standard post reads cost $0.005 per returned post. Reads are charged per resource, so one request can incur many charges.
• LinkedIn: use an authorized export, content you have permission to collect, or an approved API integration. Reading member posts is restricted. Its user agreement separately restricts automated access and copying others’ content without consent.

YouTube has separate quota buckets. Search has a default limit of 100 calls daily, at one unit per call. Comment collection uses the 10,000-unit daily bucket for endpoints without separate quotas (quota calculator). Caption downloads through its API require permission to edit the video; public comments and caption downloads are different collection tasks.

Try it: scrape a Hacker News discussion

Find a relevant Hacker News discussion and copy the numeric ID from its URL. For example, a URL ending in item?id=8863 has the ID 8863. The collector reads that item, follows its reply tree and writes one JSON record per item to a local file.

1. Install Node.js 22 or later if you do not already have it.
2. Download the ZIP linked below and extract it. Place collect-hn-thread.mjs in a research folder.
3. Open a terminal in that folder. Run: node collect-hn-thread.mjs 8863. Replace 8863 with your chosen discussion ID.
4. Open the new hn-8863 folder. items.jsonl contains the records. manifest.json records the source, collection time, missing items and whether collection completed.
5. For another discussion, run the command with its ID. To repeat a collection, supply a new output folder: node collect-hn-thread.mjs 8863 hn-8863-repeat.

Download the Hacker News collector ZIP. It uses the official API, runs sequentially, retries failed requests and keeps existing output folders intact. It saves API text, which can contain HTML, and flags unavailable or deleted content. The example ID demonstrates the command, not a finding about AI products.

For other platforms, ask AI to adapt the collection step to their approved API. Keep credentials in your local environment. Request pagination, reply traversal, rate-limit handling and a record of missing data. Test one thread before collecting a larger batch.

Step 3: Give AI the dataset

Upload the collected files and manifests to a tool that accepts them, subject to your rights to share the content. Keep the originals unchanged. If the tool cannot read everything, split the dataset into batches and retain stable IDs. Ask it to report which records it actually read.

Use labels such as first-hand report, hearsay, supported by documentation, contradicted by documentation and unverified. A model’s numerical correctness score can hide uncertainty. Ask for the evidence behind each label and verify it yourself.

Count distinct reports, not every mention in a reply chain. Several replies repeating the opening post are not several independent experiences. Write “six of the twenty collected discussions mentioned pronunciation problems,” rather than treating that count as the share of all customers affected.

Step 4: Check the findings against the sources

Open the original posts behind the findings that matter. Check whether AI preserved the context, dates and distinction between a question and an actual experience. Confirm current prices, product names and plan limits on vendor pages. Documentation can verify a feature claim without disproving someone’s reported experience of a bug.

For the avatar-video question, a useful output might group reports about pronunciation, editing and learning-system export. That is an illustrative example, not a result from a collected dataset. Your report should name the communities, collection dates, selection rules, missing content and source IDs behind its actual findings.

Correction, October 14, 2026: An earlier version generalized Claude access restrictions to AI assistants broadly and recommended manual copying without enough qualification. This version corrects those claims, clarifies API quotas and pricing, and replaces correctness scores with evidence labels. This guide provides a collection method; it does not report a completed social-media study.