[ERROR] CRITICAL:twint.run:Twint:Feed:noDataExpecting ~ Inconsistent results [High Severity]

Python 3.6
twint 2.1.7 updated from master
Have searched issues without finding anything
Running on Ubuntu 18.04, anaconda, jupyter notebook

Commands run:

import twint
import pandas as pd

c = twint.Config()
c.Pandas=True
c.Search = "#nfl"
c.Hide_output=True
c.Since = '2019-12-01'
c.Until = '2019-12-02'

twint.run.Search(c)
df = twint.storage.panda.Tweets_df

Hi, thanks for writing this package, it's very useful. I'm clearly not using it right though. I ran the commands above as a test, using"#nfl" as a query because it's innocuous and guaranteed to have a lot of results over the course of one day, but I am getting inconsistent results.

First, when I run it I get a lot of these warnings (which I saw from another issue are probably related to http/https?): CRITICAL:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0)
That's fine though, the script still runs. The problem is the results are inconsistent. I ran it last night and got back 6,832 tweets, then ran it again this morning as part of testing some other code and got 4,710 tweets. When I saw that I ran it again and got 0 tweets.

I have a couple of questions if that's okay. Is twint caching the results of queries somewhere, and if so, how do I clear the cache? Is this inconsistent behaviour expected (is it a Twitter search page thing?) and if so, does it make sense to run the same search multiple times and concatenate results? Finally, is there a suggested best practice for searching date ranges? (i.e. If you want all the tweets for a hashtag for the past 3 months, is it better to do one big search or break the search into daily or weekly time ranges?)

Again, thanks for this package. Great work.

[pielco11]

Twint does not cache results, queries or anything else. Every single piece of data is provided by Twitter. It makes sense to run multiple searches until it makes sense to you, for example you could monitor specific hashtags and see which user deletes more tweets than others. On the other side, you will most probably get duplicates if you don't filter the data with since and until (or other parameters).

If the hashtag is really popular, I'd split the searches in months if not even weeks.

Thanks. So, is it normal though that if I run the exact same search for the same query 12 hours apart that I get very different numbers of tweets back? When I ran it yesterday I got ~6800 results and when I ran it again this morning I only got ~4700. (When I ran it a third time it returned 0 tweets??)

It's not really that big a deal, I'm just hoping to understand the expected behaviour to explain to analysts when I explain the data to them. If I'm only getting a subset, or a random subset of the data, that's fine, but I need to understand that before they ask me about it.

Thanks

[pielco11]

If you run the same script over time, so with the same date-time ranges and other parameters, the dataset should be "constant" since it does not depend by the relative time which you run the script at.

So if we have a variation and what can variate is the time interval or the dataset itself, I guess that (in this case) is the dataset that's changing. Most probably due to deleted tweets, accounts that go from open to closed, accounts that get delete/suspended.

Let's say that tweets with id = 1,2,3 are sent in the interval time (A,B). If we run the scripts at C with C > B, and the tweets don't get deleted/shadowed et similia , we'll always get tweets with id = 1,2,3. The interval is closed and nothing can go it or get out by itself. The only way which a tweet can go out of that interval is just being deleted/shadowed and so on.

Thank you. I guess my strange results were a glitch. I mean, it's unlikely that 2000 tweets got deleted for that hashtag overnight. I'll keep experimenting.

Much appreciated.

[pielco11]

I can guarantee you that what Twint returns is what Twitter gives (and this can be proven, just run with --debug or config.Debug = True and in the twint-request_urls.log file you'll see every request made, and you can replicates the request with any software of your choice).

You can run the script with your handle as target, if you always get the same tweets everything is fine otherwise it needs to be investigated.

Best of luck!

Thanks, I'm sure it's something I'm doing. I will keep at it.

Sorry to come back to this, but just wanted you to be aware of the results of my testing.
I ran this exact code multiple times in a row, and each time it returned very different results. I don't know if there is something wrong with my code (please tell me if so!) or if there is something else going on.

(running with python 3.6 and fully updated twint)

c = twint.Config()
c.Hide_output=True
c.Pandas_clean=True
c.Pandas=True
c.Search="#nfl"
c.Since='2019-12-01'
c.Until='2019-12-02'

twint.run.Search(c)

I just ran that exact code 4 times in a row. It returned this many tweets:
Run #1: 1,909 tweets
Run #2: 280 tweets
Run #3: 13,207 tweets
Run #4: 3,015 tweets

I'm really not sure what to do with that at this point? Am I doing something wrong? On the three runs with the lower values, it also returned this

CRITICAL:root:twint.run:Twint:Feed:Tweets_known_error:Expecting value: line 1 column 1 (char 0)
Expecting value: line 1 column 1 (char 0) [x] run.Feed
[!] if get this error but you know for sure that more tweets exist, please open an issue and we will investigate it!

Any advice would be welcome.

Edit: For what it's worth, it's definitely something to do with the feed being disrupted. That error is being thrown in the Feed method of Twint, and it seems to happen most of the time. I re-ran that script in a loop a bunch of times, and 13, 207 tweets seems to be the actual correct number, but it doesn't come back with that very often.

[pielco11]

I've tried your query and I can confirm that the results are not consistent. That's really strange and needs to be investigated.

I'll keep you updated