Falcon API doesn't work as expected

[nspacer]

Describe the bug
For quite some time we are observing one issue with falcon API. So our organization is reporting all ec2 instances to falcon. We are doing falcon api query from python and getting all the ec2 instance that is reporting to falcon and creating one report.

The problem is the falcon api sometimes returns one instance and sometimes its not. This is very peculiar. Suppose for one 'abc' instance the api returning that it exists in falcon db on 22nd March. On 23rd March the api doesn't return anything. Again one 24th March, it is returning the instance exists. This is very confusing as to why this happens. Our Job runs eveyday at 5 AM CET.

To Reproduce
Install falcon agents on aws ec2 instance. The instance will be reported to falcon db. Query the falcon API daily and check the status if the instances are seen everyday or not. Not sure if this will happen for small instance numbers but definitely facing issues with huge numbers of ec2 instances.

Expected behavior
An AWS ec2 instance should return by the API everyday if it is not deleted or the falcon agent is not removed.

Environment (please complete the following information):

• OS: [Linux and Windows]
• Python: [3.9]
• FalconPy: [_VERSION = '1.1.6'
  _MAINTAINER = 'Joshua Hiller'
  _AUTHOR = 'CrowdStrike'
  _AUTHOR_EMAIL = '[email protected]'
  _CREDITS = 'CrowdStrike'
  _DESCRIPTION = "The CrowdStrike Falcon SDK for Python 3"
  _TITLE = "crowdstrike-falconpy"
  _PROJECT_URL = "https://github.com/CrowdStrike/falconpy"
  _DOCS_URL = "https://www.falconpy.io"
  _KEYWORDS = ["crowdstrike", "falcon", "api", "sdk", "oauth2", "devsecops", "crowdstrike-falcon"]]

Additional context
Below is the code snippet that we use.

while new_offset < total_records: response_from_falcon = falcon.query_devices_by_filter(offset=new_offset, limit=2500, sort="status.desc") responses_from_falcon_hash_ids_interim = response_from_falcon["body"]["resources"] new_offset = response_from_falcon["body"]["meta"]["pagination"]["offset"] total_records = response_from_falcon["body"]["meta"]["pagination"]["total"] strings_of_hash_ids = ','.join(responses_from_falcon_hash_ids_interim) device_details = falcon.get_device_details(ids=f"{strings_of_hash_ids}")["body"]["resources"] for instances in device_details: try: if any('instance_id' in check_for_instance for check_for_instance in instances): row = [timestamp, instances["service_provider_account_id"], instances["instance_id"], instances["os_version"], instances["product_type_desc"]] csv_rows.append(row) else: row = [timestamp, "Company", instances["hostname"], instances["os_version"], instances["product_type_desc"]] csv_rows.append(row) except KeyError as e: print('Below key is not found in response') print(e) return csv_rows

I am attaching one sample excel file which actually shows how it is returning one particular instance in a strange way. We know it for one instance but I am sure this is happening for may instances in our organization.
Book1.xlsx

[crowdstrikedcs]

Hi @nspacer thanks for the question! Converting this to a discussion to better assist.

I took a look at the attached code and wanted to take a look at the loop.

while new_offset < total_records:
    response_from_falcon = falcon.query_devices_by_filter(offset=new_offset, limit=2500, sort="status.desc")
    responses_from_falcon_hash_ids_interim = response_from_falcon["body"]["resources"]
    new_offset = response_from_falcon["body"]["meta"]["pagination"]["offset"]
    total_records = response_from_falcon["body"]["meta"]["pagination"]["total"]
    strings_of_hash_ids = ','.join(responses_from_falcon_hash_ids_interim) 
    device_details = falcon.get_device_details(ids=f"{strings_of_hash_ids}")["body"]["resources"]

While this does paginate through the list is it possible that there are more than 10000 records in your environment? Specifically the query_devices_by_filter operation can only return 10000 records whether they are active or deleted. This could result in differences based on the limited return size.

The query_devices_by_filter_scroll does not have this limitation and uses a offset token for scrolling. I've updated your loop for this new operation. Give it a try and see if you observe different results.

//set these to initialize the loop
new_offset = ""
total_records = 1

while total_records > 0:
    response_from_falcon = falcon.query_devices_by_filter_scroll(offset=new_offset, limit=2500, sort="status.desc")
    responses_from_falcon_hash_ids_interim = response_from_falcon["body"]["resources"]
    new_offset = response_from_falcon["body"]["meta"]["pagination"]["offset"]
    total_records = response_from_falcon["body"]["meta"]["pagination"]["total"]
    strings_of_hash_ids = ','.join(responses_from_falcon_hash_ids_interim) 
    device_details = falcon.get_device_details(ids=f"{strings_of_hash_ids}")["body"]["resources"]

[nspacer]

Dear @crowdstrikedcs , thanks for your reply. Below are my observations.

1. query_devices_by_filter works fine for me with my pagination method. It returns all the 28579 records but at each time 2500 limit as I have mentioned in the filter. It might have to go through 10 times or more through the loop but it returns all the records. The problem is sometimes it doesn't return couple of instances which already have falcon installed.
2. This while loop you have mentioned will go into an infinite loop. As all the time total_records will come as total number of records in the API which is 28579.
3. I am getting below error with

falcon.query_devices_by_filter_scroll(offset=new_offset, limit=2500, sort="status.desc")

_**response_from_falcon: {'status_code': 400, 'headers': {'Server': 'nginx', 'Date': 'Mon, 03 Apr 2023 12:50:03 GMT', 'Content-Type': 'application/json', 'Content-Length': '192', 'Connection': 
'keep-alive', 'Content-Encoding': 'gzip', 'Strict-Transport-Security': 'max-age=15724800; includeSubDomains, max-age=31536000; includeSubDomains', 'X-Cs-Region': 'us-1', 'X-Cs-Traceid': 'f0f
716cc-4da6-43f5-85b2-bd3e0d725057', 'X-Ratelimit-Limit': '6000', 'X-Ratelimit-Remaining': '5999'}, 'body': {'meta': {'query_time': 0.003417067, 'powered_by': 'device-api', 'trace_id': 'f0f71
6cc-4da6-43f5-85b2-bd3e0d725057'}, 'resources': [], 'errors': [{'code': 400, 'message': 'Bad request'}]}}**_

I am using below code snippet

total_records = 1
new_offset = 0
#while new_offset <= total_records:
while total_records > 0:
    response_from_falcon = falcon.query_devices_by_filter_scroll(offset=new_offset, limit=2500,
                                                                sort="status.desc")
    #response_from_falcon = falcon.query_devices_by_filter(offset=new_offset, limit=2500, sort="status.desc")
    print(f"response_from_falcon: {response_from_falcon}")
    responses_from_falcon_hash_ids_interim = response_from_falcon["body"]["resources"]
    print(f"new_offset: {new_offset} and total_records: {total_records}")
    new_offset = response_from_falcon["body"]["meta"]["pagination"]["offset"]
    total_records = response_from_falcon["body"]["meta"]["pagination"]["total"]
    print(f"new_offset after modifications: {new_offset} and total_records after modifications: {total_records}")
    strings_of_hash_ids = ','.join(responses_from_falcon_hash_ids_interim)
    device_details = falcon.get_device_details(ids=f"{strings_of_hash_ids}")["body"]["resources"]

[crowdstrikedcs]

Hi @nspacer WRT to your first, I misspoke earlier about only being able to pull 10000 records, this does not apply to customers in the US-1 cloud currently where the limit is 150000. I would still encourage usage of the query_devices_by_filter_scroll operation as due to its difference in using an offset pointer we will have better results scrolling through responses with more than 10000 records.

To clarify on your second point, with the query_devices_by_filter_scroll operation, after having reached the last page of data the following response will set the total to zero indicating no more records are to be returned, at which point the loop will exit. This is a difference from the query_devices_by_filter operation where the total changes during pagination. In my testing the code I posted earlier will exit the loop.

For your third point, the error appears to be caused by the value of new_offset being instantiated to 0. This should be an empty string at first as the offset parameter for the operation is expected to be a string as well. I have updated the code you have posted to reflect this

total_records = 1
new_offset = ""
#while new_offset <= total_records:
while total_records > 0:
    response_from_falcon = falcon.query_devices_by_filter_scroll(offset=new_offset, limit=2500,sort="status.desc")
    #response_from_falcon = falcon.query_devices_by_filter(offset=new_offset, limit=2500, sort="status.desc")
    print(f"response_from_falcon: {response_from_falcon}")
    responses_from_falcon_hash_ids_interim = response_from_falcon["body"]["resources"]
    print(f"new_offset: {new_offset} and total_records: {total_records}")
    new_offset = response_from_falcon["body"]["meta"]["pagination"]["offset"]
    total_records = response_from_falcon["body"]["meta"]["pagination"]["total"]
    print(f"new_offset after modifications: {new_offset} and total_records after modifications: {total_records}")
    strings_of_hash_ids = ','.join(responses_from_falcon_hash_ids_interim)
    device_details = falcon.get_device_details(ids=f"{strings_of_hash_ids}")["body"]["resources"]

Let me know how this works

[nspacer]

Dear @crowdstrikedcs , this is working and generating similar response like the other method till now. I do have one query still:

You said the offset can live for 2 minutes and after that new offset will be generated. Now think of a scenario where the job runs for more than 2 mins and the offset expire. Do then the data will be taken up after the last offset or it will be reset all over again. I am not sure if I am clear on my question or not. All I want to confirm about the last indexed data that is picked up by the job before the offset expiry. When the new offset will be created will it pick up from next index or not.

Let me know if this explain the question or my understanding of the offset expiry is wrong.

[crowdstrikedcs]

Hi @nspacer good news! Try that out for a while and let us know if you continue to run into issues. With this endpoint each request for a page extends the window by 2 minutes. So you have 2 minutes from the previous request before the context is expired. Additionally in the pagination object in the response contains a expires_at field which contains the expiry time as an epoch timestamp. You'll notice this value increases with each request.

[nspacer]

Dear @crowdstrikedcs , after checking the data for couple of days, we can see that the resources that are short lived do not appear in the API. So what we are observing is, suppose one of the instance which is started for 2 mins and then stopped again after 2 mins, do not appear in the API. But is available in the snapshot taken directly from crodstrike website.

Do you know any reason for this behaviour?

[crowdstrikedcs]

Hi @nspacer interesting, I wouldn't expect this to case. For these assets are you sure the agent is being installed correctly? Are you using any filter when querying for devices?

[nspacer]

@crowdstrikedcs , yes these instances are crowdstrike installed already. The reason why I am certain is that, when we take the dump directly from the crowdstrike website all data appear. Only when we use the API, these instances are missing.