Smart Crawling: A New Approach toward Focus Crawling from Twitter - Laboratoire Interdisciplinaire des Sciences du Numérique Access content directly
Preprints, Working Papers, ... Year : 2021

Smart Crawling: A New Approach toward Focus Crawling from Twitter


Twitter is a social network that offers a rich and interesting source of information challenging to retrieve and analyze. Twitter data can be accessed using a REST API. The available operations allow retrieving tweets on the basis of a set of keywords but with limitations such as the number of calls per minute and the size of results. Besides, there is no control on retrieved results and finding tweets which are relevant to a specific topic is a big issue. Given these limitations, it is important that the query keywords cover unambiguously the topic of interest in order to both reach the relevant answers and decrease the number of API calls. In this paper, we introduce a new crawling algorithm called "SmartTwitter Crawling" (STiC) that retrieves a set of tweets related to a target topic. In this algorithm, we take an initial keyword query and enrich it using a set of additional keywords that come from different data sources. STiC algorithm relies on a DFS search in Twittergraph where each reached tweet is considered if it is relevant with the query keywords using a scoring, updated throughout the whole crawling process. This scoring takes into account the tweet text, hashtags and the users who have posted the tweet, replied to the tweet, been mentioned in the tweet or retweeted the tweet. Given this score, STiC is able to select relevant tweets in each iteration and continue by adding the related valuable tweets. Several experiments have been achieved for different kinds of queries, the results showedthat the precision increases compared to a simple BFS search.
2110.06022 (2).pdf (783.64 Ko) Télécharger le fichier
Origin Files produced by the author(s)

Dates and versions

hal-04421731 , version 1 (28-01-2024)



Ahmad Khazaie, Nacéra Bennacer Seghouani, Francesca Bugiotti. Smart Crawling: A New Approach toward Focus Crawling from Twitter. 2024. ⟨hal-04421731⟩
9 View
3 Download



Gmail Mastodon Facebook X LinkedIn More