Twint - An Advanced Twitter Scraping Together With Osint Tool

Formerly known equally Tweep, Twint is an advanced Twitter scraping tool written inwards Python that allows for scraping Tweets from Twitter profiles without using Twitter's API.

Twint utilizes Twitter's search operators to permit y'all scrape Tweets from specific users, scrape Tweets relating to sure as shooting topics, hashtags & trends, or form out sensitive information from Tweets similar e-mail too telephone numbers. I uncovering this real useful, too y'all tin become actually creative amongst it too.

Twint also makes especial queries to Twitter allowing y'all to also scrape a Twitter user's followers, Tweets a user has liked, too who they follow without whatsoever authentication, API, Selenium, or browser emulation.

Benefits
Some of the benefits of using Twint vs Twitter API:

• Can fetch almost all Tweets (Twitter API limits to final 3200 Tweets only)
• Fast initial setup
• Can live used anonymously too without Twitter sign up
• No charge per unit of measurement limitations

Requirements

• Python 3.6

Installing
Git:

git clone https://github.com/twintproject/twint.git pip3 install -r requirements.txt

Pip:

pip3 install --upgrade -e git+https://github.com/twintproject/twint.git@origin/master#egg=twint

Pipenv:

pipenv install -e git+https://github.com/twintproject/twint.git#egg=twint

CLI Basic Examples too Combos
Influenza A virus subtype H5N1 few elementary examples to assist y'all empathize the basics:

• twint -u username - Scrape all the Tweets from user's timeline.
• twint -u username -s pineapple - Scrape all Tweets from the user's timeline containing pineapple.
• twint -s pineapple - Collect every Tweet containing pineapple from everyone's Tweets.
• twint -u username --year 2014 - Collect Tweets that were tweeted before 2014.
• twint -u username --since 2015-12-20 - Collect Tweets that were tweeted since 2015-12-20.
• twint -u username -o file.txt - Scrape Tweets too relieve to file.txt.
• twint -u username -o file.csv --csv - Scrape Tweets too relieve equally a csv file.
• twint -u username --email --phone - Show Tweets that powerfulness convey telephone numbers or electronic mail addresses.
• twint -s "Donald Trump" --verified - Display Tweets past times verified users that Tweeted close Donald Trump.
• twint -g="48.880048,2.385939,1km" -o file.csv --csv - Scrape Tweets from a radius of 1km some a house inwards Paris too export them to a csv file.
• twint -u username -es localhost:9200 - Output Tweets to Elasticsearch
• twint -u username -o file.json --json - Scrape Tweets too relieve equally a json file.
• twint -u username --database tweets.db - Save Tweets to a SQLite database.
• twint -u username --followers - Scrape a Twitter user's followers.
• twint -u username --following - Scrape who a Twitter user follows.
• twint -u username --favorites - Collect all the Tweets a user has favorited.
• twint -u username --following --user-full - Collect total user information a individual follows
• twint -u username --profile-full - Use a slow, but effective method to assemble Tweets from a user's profile (Gathers 3200 Tweets, Including Retweets).
• twint -u username --retweets - Use a quick method to assemble the final 900 Tweets (that includes retweets) from a user's profile.
• twint -u username --resume 10940389583058 - Resume a search starting from the specified Tweet ID.

More item close the commands too options are located inwards the wiki

Module Example
Twint tin right away live used equally a module too supports custom formatting. More details are located inwards the wiki

import twint  # Configure c = twint.Config() c.Username = "noneprivacy" c.Search = "#osint" c.Format = "Tweet id: {id} | Tweet: {tweet}"  # Run twint.run.Search(c)

Output

955511208597184512 2018-01-22 18:43:19 GMT pineapples are the best fruit

import twint  c = twint.Config()  c.Username = "noneprivacy" c.Custom["tweet"] = ["id"] c.Custom["user"] = ["bio"] c.Limit = 10 c.Store_csv = True c.Output = "none"  twint.run.Search(c)

Storing Options

• Write to file
• CSV
• JSON
• SQLite
• Elasticsearch

Elasticsearch Setup
Details on setting upwards Elasticsearch amongst Twint is located inwards the wiki.

Graph Visualization

Graph details are also located inwards the wiki.
We are developing a Twint Desktop App.

FAQ

I tried scraping tweets from a user, I know that they be but I'm non getting them

Twitter tin shadow-ban accounts, which agency that their tweets volition non live available via search. To solve this, dice --profile-full if y'all are using Twint via CLI or, if are using Twint equally module, add together config.Profile_full = True. Please banker's complaint that this procedure volition live quite slow.

More Examples

Followers/Following

To become solely follower usernames/following usernames

twint -u username --followers
twint -u username --following

To become user information of followers/following users

twint -u username --followers --user-full
twint -u username --following --user-full

userlist

To become solely user information of user

twint -u username --user-full

To become user information of users from a userlist

twint --userlist inputlist --user-full

Contact
If y'all convey whatsoever questions, desire to bring together inwards on discussions, or require extra help, y'all are welcome to bring together our Twint focused Slack server.
If y'all are interested inwards OSINT too withal seeking for assist or suggestions, bring together the OSINT community at OSINT Team (there is a specific Twint channel)

Download Twint