# Collecting X (Twitter) data for research > Build Twitter/X research datasets in Python: paginate search and timelines, batch profile lookups, cost per million tweets, notes on terms and ethics. HTML page: https://www.twittrapi.com/use-cases/research Solutions · Research The free academic access to the Twitter API ended in 2023. TwittrAPI is a third-party API for public X data, billed per call: $0.065 to $0.09 per 1,000, prepaid, with a balance that does not expire. This page shows how to build a dataset with it, what it costs and how long it takes. [Measure your dataset](https://www.twittrapi.com/start) [Pagination guide →](https://www.twittrapi.com/docs/helpful-infos/cursor) $0.10 free credit after sign-up: run a few pages of your query before you budget. Before you collect ## Terms, ethics and scope TwittrAPI is not affiliated with X Corp. X's terms restrict automated collection of its content; our article [How to scrape Twitter (X) data](https://www.twittrapi.com/blog/how-to-scrape-twitter-x-data) summarizes them with sources. Whether your project should use the official API or a third-party source is a decision for your institution, not for us. - Public data only: protected accounts, DMs and deleted tweets are not available. - Tweets are personal data under GDPR and similar laws, even when public. Ethics boards usually want a data management plan: what you collect, how you pseudonymize authors, how long you keep it, who has access. - Collect less. The simple format already drops most of X's payload; store `id`, `createdAt`, `text`, `lang` and the author ID if that is all the analysis needs. - Many venues accept datasets of tweet IDs rather than full text. Check your venue's policy. Endpoints ## Which endpoint for which dataset Every row is a GET with your key. Paginated endpoints return one page per call and a `nextCursor` in the simple format. - Tweets matching a query in a time window /search?category=Latest One page; since: and until: in the query - What an account posted (recent) /user/{userId}/tweets, /user/{userId}/tweetsAndReplies One page - Reply trees /tweet/{tweetId} One page of reply threads - Quote networks /tweet/{tweetId}/quotes One page - Follower graphs /user/{userId}/followers, /user/{userId}/followings One page - Profiles for a list of IDs /user/users/by/ids Up to 100 profiles - Handles to IDs /user/usernames/to/ids Up to 100 handles - Does this handle still exist? /user/usernames/status Up to 500 handles - A community or a curated list /communities/{communityId}/tweets, /lists/tweets One page The right column is what one call returns. Profile timelines end where X stops showing older tweets. For older tweets of one account, search `from:username since:… until:…` in windows. Every endpoint: [reference](https://www.twittrapi.com/docs/reference). Python ## A search window to JSON Lines With the simple format every page carries `nextCursor` and `hasNextPage`, so the loop has a clear end. The SDK's `paginate` follows the cursor for you and retries 429 and 5xx with backoff. - `count` is a hint (1 to 100); X decides the real page size. - Cursors are X's and can expire after a while. If a run can be interrupted, store the last cursor with your progress and resume soon. - Calls that fail with 400, 408, 409 or a 5xx error are refunded; 429 is not charged. 1. 01 Pick a query and one day The search window is the query plus a since: and until: date. 2. 02 Page through it with the SDK Up to 100 tweets per page; the SDK follows the cursor until the last page. 3. 03 Write one line per tweet ID, time, language, author ID and text to a JSON Lines file. Deleted or protected tweets are skipped. 4. 04 Count the pages Pages fetched are the calls you pay for, so every run reports its own cost. [Python collector](https://www.twittrapi.com/docs/recipes#collect-a-search-window-to-json-lines) Throughput ## Going faster: split the time range Pagination is sequential: page 2 needs the cursor from page 1. At roughly a second per response, one cursor chain gives about one page per second, however high your rate limit is. Split the range into windows (a day, an hour) and run the windows in parallel. With the 30 requests/s included after the first top-up, 25 workers stay under the limit. Higher limits up to 1,000 requests/s are monthly add-ons you can book for the month of the crawl and cancel after. [Rate limits](https://www.twittrapi.com/docs/helpful-infos/rate-limits) Thirty days of one query become thirty independent windows. With 25 workers, 25 cursor chains run at once, and each writes its own file. [Parallel example](https://www.twittrapi.com/docs/recipes#going-faster-split-the-time-range) Cost ## What a dataset costs Calls = items you need ÷ items per call. The page sizes below are assumptions for illustration; measure yours with a few pages first. ### 1,000,000 tweets via search $4.50 at Standard **Assumption** 20 tweets per page **Calls** 50,000 **At Scale** $3.25 ### 1,000,000 tweets via search $0.90 at Standard **Assumption** 100 tweets per page **Calls** 10,000 **At Scale** $0.65 ### 1,000,000 profiles by ID $0.90 at Standard **Assumption** 100 per call (batch endpoint) **Calls** 10,000 **At Scale** $0.65 ### Followers of an account with 500,000 followers $0.45 at Standard **Assumption** 100 per page **Calls** 5,000 **At Scale** $0.33 ### Reply threads of 10,000 tweets $0.90 at Standard **Assumption** 1 page each **Calls** 10,000 **At Scale** $0.65 Prices per 1,000 calls: Standard $0.09, Scale $0.065. Refunded calls (400, 408, 409 or a 5xx error) are not in these numbers; empty pages at the end of a window are. Time: 50,000 sequential pages at about one per second is about 14 hours; in 25 parallel windows it is well under an hour. **Budgeting a funded project.** Top up once: $100 gets Volume ($0.075 per 1,000) for 30 days, $500 within 30 days gets Scale ($0.065). The balance stays until you use it. Crypto has no surcharge; card top-ups carry a 9 % card fee plus VAT. If your institution needs an invoice, [request card payment with invoice](https://app.twittrapi.com/top-up#business) in the dashboard; it is enabled after a short review. Methods ## Reproducibility - Log the exact query, the window, the format and the date of collection with each file. Simple-format responses carry `X-Response-Format: simple`, so you can record which format you got. - Simple-format fields are stable: new fields may be added, existing ones are not renamed or removed. - Search results depend on X's index on the day you ask. Two runs a month apart can differ; record when you collected. - Keep a raw-format sample if you may need fields the simple format leaves out (cards, polls, edit history, community notes). FAQ ## Research questions **Is there an academic tier?** There is no separate tier. New accounts get $0.10 free credit (about 1,100 calls at Standard), enough to measure page sizes and costs for your own queries before you budget. **Can I get historical tweets from years ago?** Search accepts since: and until:, so you can query past windows. How much X returns for older periods varies; test your window with a few pages first. **Can I share the dataset?** Check your venue's policy and X's terms. Sharing tweet IDs instead of full text is common practice; others can look the tweets up again. **Is it legal to use this for research?** TwittrAPI provides public data and is not affiliated with X Corp. Whether a third-party source fits your project is for your institution's legal and ethics offices to decide; our scraping guide summarizes X's terms with sources. **How do I pay from a grant?** Top up once and spend over time; the balance does not expire. Crypto has no surcharge. If you need an invoice, request card payment with invoice in the dashboard; it is enabled after a short review. **Does it work in Jupyter or R?** Yes. In Jupyter use the Python SDK; the collector recipe in the docs runs there unchanged. From R or any other language it is plain HTTPS: GET with the X-API-KEY header and format=simple in the query string. Related ## Read next - [Pagination and cursors → How cursors work, when they expire /docs/helpful-infos/cursor](https://www.twittrapi.com/docs/helpful-infos/cursor) - [Simple format → Fields, schema and stability /docs/simple-format](https://www.twittrapi.com/docs/simple-format) - [Search reference → Operators, since:/until:, filters /docs/reference/search](https://www.twittrapi.com/docs/reference/search) - [How to scrape Twitter (X) data → Options and X's terms, with sources /blog/how-to-scrape-twitter-x-data](https://www.twittrapi.com/blog/how-to-scrape-twitter-x-data) - [TwittrAPI vs the official X API → When the official API fits better /compare/official-x-api](https://www.twittrapi.com/compare/official-x-api) - [AI agents → LLM coding of tweets with tool calls /use-cases/ai-agents](https://www.twittrapi.com/use-cases/ai-agents) ## Measure your dataset before you budget it Run a few pages of your query with the $0.10 free credit, count the tweets per page, and the cost table above becomes your own. [Get an API key](https://www.twittrapi.com/start) [Pricing →](https://www.twittrapi.com/pricing)