Multi-Tenant RTR

[RoemIko]

Is there a way to authenticate as an MSSP to mutliple CID's and perform RTR commands in every child?
At the moment I am authenticating per tenant with 'member_cid', however i got the 'API limit exceeded'. after trying to authenticate with over 100 CID's

[jshcodes]

Hi @RoemIko -

Thank you for the excellent question!

Sending RTR commands to a large number of hosts within the same CID is straightforward, but for hosts across large numbers of CIDs, this is definitely more complex. At the moment, you are running into the rate limit for the tokens operation when authenticating to subsequent CIDs. Since this is a hard limit, we'll have to work within this restriction.

I'd like to answer this using a sample. I'm working on this today and will post the link as soon as it's merged.

[RoemIko]

Thank you! I had been looking for examples, but couldn't find them. Looking forward to the example you write.

[RoemIko]

@jshcodes is there an update on this question?

[jshcodes]

Still in flight! There will be a few pieces to this example, and I had a couple experiments to run. It's on the way, but while you wait you're welcome to continue to ask additional questions. 😄

[jshcodes]

Hi @RoemIko!

These samples are in the process of being reviewed now. I'm hoping to have them merged to main and the response to your question posted later today. Apologies for the delay. 😅

[RoemIko]

Hi @jshcodes,

Thats fine! Looking forward for the examples 😄

[jshcodes]

Hi @RoemIko –
 
Whenever we perform large numbers of operations against the API, especially token operations, rate limits are something we must keep in mind. Depending on our scenario requirements, and the pattern we’ve implemented, we have a few options for addressing this.

Warning: This answer is a little verbose.
 
To try and scope this response a bit so it’s easier to focus on our discussion points, I’m going to define our problem statement with the following parameters:

• We have many child CIDs each with a differing number of registered endpoints of varying types.
• We need to select the hosts we wish to target in each child tenant using a simple FQL filter.
• To avoid unnecessary complexity that is unrelated to the topic we’re discussing; we will use the QueryDevicesByFilter operation without pagination to retrieve host AIDs. This means our sample will be limited to executing a command against a maximum of 5,000 endpoints within each tenant.
• We need to enable offline queueing for RTR commands, and log the cloud request IDs for commands that are queued by CID / device ID.
• We will need to implement a simple rate limit handler for our one child authentication operation.

We’re leveraging the Flight Control Service Class and have a large list of child tenants. We’ve been tasked with looping through each tenant to process commands against hosts within. When we first look at this problem, it can be tempting to loop through the entire list to create our authenticated Service Class objects first, as this allows for some pretty neat programmatic patterns. In lots of scenarios, the pattern works and is easy to read.

A simple example that will potentially rate limit

# Connect to the Hosts Service Class for every child CID in our list
connections = {}
for child_id in list_of_children:
    connections[child_id] = Hosts(client_id=”XYZ”,
                                  client_secret=”XYZ”,
                                  member_cid=child_id
                                  )

This code produces a dictionary, keyed by child CID, containing a properly authenticated Hosts Service Class object for that specific CID only. This is relatively elegant, allowing us to interact with all children via the one dictionary object using simple logic.

for child_id, hosts in connections.items():
    print(f”Retrieving results for {child_id}”)
    result = hosts.query_devices_by_filter()

Another cool variation

One of the things I like about Python is all the different ways you can approach (and solve) a problem. For discussion purposes, let’s pretend the actions we want to take against our child hosts require additional API service collections and expand on our previous example a bit to add more dictionary functionality.

# Connect to the Hosts, RTR and Sensor Download Service Classes for every child CID in our list
connections = {}
for child_id in list_of_children:
    connections[child_id] = {
        “hosts”: Hosts(client_id=”XYZ”,
                       client_secret=”XYZ”,
                       member_cid=child_id
                       ),
        “rtr”: RealTimeResponse(client_id=”XYZ”,
                                client_secret=”XYZ”,
                                member_cid=”XYZ”
                                ),
        “sensor”: SensorDownload(client_id=”XYZ”,
                                 client_secret=”XYZ”,
                                 member_cid=”XYZ”
                                 )
    }

Now we have a dictionary, again keyed by child CID, but we have several Service Classes available to us all in the one place. This results in another simple and easy to read pattern for implementing the process we want to perform within each child tenant.

# Using the results from our variation
for child_id, sdk in connections.items():
    print(f”Retrieving results for {child_id}”)
    hosts_result = sdk[“hosts”].query_devices_by_filter()
    sensor_result = sdk[“sensor”].get_sensor_installer_ccid()
    # Etc…

Unfortunately, due to how often authentication occurs, both patterns can run into rate limits for new tokens if we’re not careful. Token operations maintain different rate limits, that are tracked separately from the general rate limit, which means we can hit this upper bound faster when working with them.

In our first example, we loop through our list of child CIDs, creating an instance of the Hosts Service Class first, before performing any desired action within the tenant. Each time we do this, we authenticate to the API. Depending on the speed of our machine, available bandwidth, and what else is going on in our code, we can hit rate limits for new tokens just processing this synchronously within a few minutes.

Our second example compounds this problem, adding in additional Service Classes we want to have available when we perform our tenant actions. Now we’re not making a single authentication request for each child tenant in a short amount of time, we’re making three! We’re even more likely to hit rate limits in this scenario.

Reducing authentication requests

The first thing we can address, is the number of times we authenticate to the API and generate a new token. We expanded the dictionary above to add new functionality to our process, and while we will have to authenticate once per child CID we’re handling, thanks to Object Authentication, we do not have to do this more than once.

Object Authentication allows us to create an instance of any Service Class and then leverage the auth_object attribute of this newly created Service Class object to authenticate and create instances of additional Service Class objects. You may also create an instance of the OAuth2 Service Class, and pass this as the auth_object keyword to new Service Classes. Both methods result in configuration parameters, including tokens, being used for subsequent object creation. No additional authentication requests to the API are performed when a Service Class is created using these methods.

Simple examples of Object Authentication

We have several examples in the samples library that demonstrate Object Authentication, but the simplest examples would be:

hosts = Hosts(client_id=”XYZ”, client_secret=”XYZ”)
rtr = RealTimeResponse(auth_object=hosts.auth_object)

or

auth = OAuth2(client_id=”XYZ”, client_secret=”XYZ”)
hosts = Hosts(auth_object=auth)
rtr = RealTimeResponse(auth_object=auth)

You do not have to create an instance of the OAuth2 Service Class, as shown in the second example immediately above. This is created for you when you create an instance of a Service Class. With this in mind, there are some scenarios where first creating an instance of the OAuth2 Service Class makes programmatic sense. For example, let’s leverage Object Authentication to clean up the number of requests to the API made when we create our expanded dictionary of Service Classes above.

# Connect to the Hosts, RTR and Sensor Download Service Classes for every child CID in our list
connections = {}
for child_id in list_of_children:
    auth = OAuth2(client_id=”XYZ”,
                  client_secret=”XYZ”,
                  member_cid=child_id
                  )
    connections[child_id] = {
        “hosts”: Hosts(auth_object=auth),
        “rtr”: RealTimeResponse(auth_object=auth),
        “sensor”: SensorDownload(auth_object=auth)
    }

Now our code is easier to read, and we have access to all three API service collections, all while leveraging the same token and only performing one API request!

With this small change, we may potentially have solved our problem. The rate limit refreshes every minute, so if we’re careful about how we implement our scenario, and if our requirements allow for it, encountering this limitation is something we can possibly avoid simply.

Handling the list with a single thread

Even when working with large batches of CIDs, if we are processing our list synchronously, we can take our desired action(s) within the child per iteration and are unlikely to hit the token rate limit. The time it takes to perform the desired non-token operations offsets time that would count against us if we were instead interacting with token operations, so we consistently stay under our maximum number of token operations per minute. This isn’t to say you can’t hit rate limits using this pattern, just that it would require the actions you’re taking within the child to occur very quickly.

This works great for most “fast” needs, but for situations where our desired action (or process) can be long running, this may be less than ideal. If there are more than 95 children, and our action requires approximately 15 minutes, our synchronous solution will be executing for 24 hours or more.

Multithreading

This is usually about the time we add multithreading into our solution, and suddenly everything is executing much faster now… so fast in fact, inside of a few minutes, we’re generating errors again. By introducing asynchronous processing into the equation, our colliding against the rate limit boundary is much more likely; we no longer benefit from the built in “wait” from the processing of the previous child’s non-token operations.

This is when we need to pay attention to the status codes (status_code) returned with each API result, specifically, status code 429. When we encounter a 429 (rate limit exceeded), we can back off our processing for a few seconds to allow the rate limit to expire and then retry the operation we were working on. If on our second try, we are still hitting the rate limit, we wait a few more seconds (but slightly longer) and try again. Continue this cycle for a static number of retries, (typically somewhere between 3 to 5 times) and then issue an error if you’re still unable to perform the request after you’ve hit the retry maximum. A variation of this solution leverages the additional rate limit headers present within the response (X-Ratelimit-Remaining) to wait for the time specified within the header. Either solution will need the ability to “retry”, as due to the nature of asynchronous processing, there are no guarantees that this process is the only one making a request at the same time.

Simple Rate limit handling example

def connect_to_hosts():
    """Return an authenticated Hosts Service Class object."""
    return Hosts(client_id=CLIENT_ID,
                 client_secret=CLIENT_SECRET
                 )


# Create an instance of the Hosts Service Class.
# Add a rate limit check here to back us off if we're
# hitting our maximum requests per minute
max_attempts = 3
attempt = 0
while True:
    attempt += 1
    hosts = connect_to_hosts()
    if hosts.token_status == 429:
        if attempt > max_attempts:
            break  # Or raise an error
        print("Rate limit met, pausing")
        time.sleep(2*attempt)
    else:
        # Break out on success and non-rate limit errors
        break

Samples discussing this topic

As part of researching for this response, I’ve put together a couple of samples based on my experiments.

• MultiCID Batch Command Executor – The first sample will accept your parent API credentials, a host FQL filter, and a desired command. It will then look up every child associated with the parent and execute the command against every host within each tenant that match the provided FQL filter. For hosts that are offline, the command is queued. Results are displayed in the terminal and output to a folder you can specify at runtime. Queued cloud requests, the associated CID and device ID are also stored within this folder in separate JSON files.

usage: multicid.py [-h] [-k FALCON_CLIENT_ID] [-s FALCON_CLIENT_SECRET] [-m] [-d OUTPUT_FOLDER] [-f FILTER] [-o SORT] [-l LIMIT] [-c COMMAND] [-t TIMEOUT] [-n NUMBER_OF_THREADS] [-x]

Execute a single RTR command across multiple hosts within multiple child tenants.

 _______ __            ______                         __ _______ __         __ __
|_     _|  |--.-----. |      |.----.-----.--.--.--.--|  |     __|  |_.----.|__|  |--.-----.
  |   | |     |  -__| |   ---||   _|  _  |  |  |  |  _  |__     |   _|   _||  |    <|  -__|
  |___| |__|__|_____| |______||__| |_____|________|_____|_______|____|__|  |__|__|__|_____|

     ___ ___       __ __   __  _______ ___ ______
    |   Y   .--.--|  |  |_|__||   _   |   |   _  \
    |.      |  |  |  |   _|  ||.  1___|.  |.  |   \
    |. \_/  |_____|__|____|__||.  |___|.  |.  |    \
    |:  |   |                 |:  1   |:  |:  1    /
    |::.|:. |                 |::.. . |::.|::.. . /
    `--- ---'                 `-------`---`------'

 ______  _______ _______ _______ _     _
 |_____] |_____|    |    |       |_____|
 |_____] |     |    |    |_____  |     |

            _______  _____  _______ _______ _______ __   _ ______
            |       |     | |  |  | |  |  | |_____| | \  | |     \
            |_____  |_____| |  |  | |  |  | |     | |  \_| |_____/

                        _______ _     _ _______ _______ _     _ _______  _____   ______
                        |______  \___/  |______ |       |     |    |    |     | |_____/
                        |______ _/   \_ |______ |_____  |_____|    |    |_____| |    \_

                                                                    FalconPy v1.1.5

optional arguments:
  -h, --help            show this help message and exit
  -k FALCON_CLIENT_ID, --falcon_client_id FALCON_CLIENT_ID
                        CrowdStrike Falcon API Client ID
  -s FALCON_CLIENT_SECRET, --falcon_client_secret FALCON_CLIENT_SECRET
                        CrowdStrike Falcon API Client Secret
  -m, --multithread     Leverage multiprocessing when executing the demonstration
  -d OUTPUT_FOLDER, --output_folder OUTPUT_FOLDER
                        Folder to output saved results
  -f FILTER, --filter FILTER
                        FQL string to use to limit target hosts. (Defaults to all Windows hosts.)
  -o SORT, --sort SORT  FQL string to use to sort returned host results.
  -l LIMIT, --limit LIMIT
                        Number of hosts to return per CID. (Maximum: 5000)
  -c COMMAND, --command COMMAND
                        Command to execute across all targeted hosts. (Defaults to return environment details.)
  -t TIMEOUT, --timeout TIMEOUT
                        Batch execution timeout in seconds. (Defaults to 120.)
  -n NUMBER_OF_THREADS, --number_of_threads NUMBER_OF_THREADS
                        Number of threads to spawn, ignored when not multithreaded. Not required.
  -x, --script_execution
                        Executes the command in raw format using runscript.(Defaults to regular execution.)

• Get RTR Result – Returns the results for previously queued RTR commands. This sample can be executed in one of three ways:
  • You can provide credentials that include a member_cid (if necessary) and a cloud request ID. The results of this cloud request are then displayed to the terminal.
  • You can provide credentials that do not include a member_cid and provide the location of the output queue file created by the MultiCID Batch Command Executor sample. The contents of the file will be parsed, and then each cloud request will be looked up, with the result (pending or completed) displayed to the terminal in order.
  • You can provide credentials that do not include a member_cid and provide the location of the output folder specified to the MultiCID Batch Command Executor sample. The contents of the folder will be parsed, and then each cloud request will be looked up, with the result (pending or completed) displayed to the terminal in order by CID.

usage: get_rtr_result.py [-h] -k FALCON_CLIENT_ID -s FALCON_CLIENT_SECRET [-m MEMBER_CID] [-b BASE_URL] [-c CLOUD_REQUEST_ID] [-q SEQUENCE] [-f QUEUE_FILE_FOLDER]

Retrieve the results of a command executed via Real Time Response.

 _______             __  _______ __
|   _   .-----.---.-|  ||       |__.--------.-----.
|.  l   |  -__|  _  |  ||.|   | |  |        |  -__|
|.  _   |_____|___._|__|`-|.  |-|__|__|__|__|_____|
|:  |   |                 |:  |
|::.|:. |                 |::.|
`--- ---'                 `---'
             _______
            |   _   .-----.-----.-----.-----.-----.-----.-----.
            |.  l   |  -__|__ --|  _  |  _  |     |__ --|  -__|
            |.  _   |_____|_____|   __|_____|__|__|_____|_____|
            |:  |   |           |__|
            |::.|:. |                       FalconPy v1.1
            `--- ---'

optional arguments:
  -h, --help            show this help message and exit
  -k FALCON_CLIENT_ID, --falcon_client_id FALCON_CLIENT_ID
                        CrowdStrike Falcon API Client ID
  -s FALCON_CLIENT_SECRET, --falcon_client_secret FALCON_CLIENT_SECRET
                        CrowdStrike Falcon API Client Secret
  -m MEMBER_CID, --member_cid MEMBER_CID
                        Child CID for MSSP scenarios
  -b BASE_URL, --base_url BASE_URL
                        CrowdStrike Base URL (Only required for GovCloud: usgov1)
  -c CLOUD_REQUEST_ID, --cloud_request_id CLOUD_REQUEST_ID
                        Cloud Request ID to retrieve, accepts comma-delimited lists
  -q SEQUENCE, --sequence SEQUENCE
                        Command result sequence ID, defaults to 0
  -f QUEUE_FILE_FOLDER, --queue_file_folder QUEUE_FILE_FOLDER
                        Load a directory of save files or a single save file for processing

Let us know if you have any questions!