ENH: Sampling a contiguous block from a DF

[YM2132]

Feature Type

• Adding new functionality to pandas

• Changing existing functionality in pandas

• Removing existing functionality in pandas

Problem Description

I wish I could use Pandas to return a random block of data from a DF.

Like .sample() but the data returned is contiguous, in essence the method randomly samples 1 row idx and then samples the next N from that point.

Feature Description

Add a contiguous param to .sample or create a new .sample_contgiuous

Draw from distribution a single idx

and select N around that as a contiguous block

Alternative Solutions

def random_contiguous_sample(df, n):
max_start = len(df) - n

start_idx = np.random.randint(0, max_start + 1)
return df.iloc[start_idx:start_idx + n]

Additional Context

No response

[gautamvarmadatla]

Hi @YM2132,

This would be really handy. I have wanted “.sample() but as a contiguous window” a few times ( for traaining purposes )

Based off your idea, my thought is that a new method like sample_window or sample_contiguous might be easier to land than adding contiguous= to .sample(). .sample() already has a lot of knobs like weights, replace, frac, and axis, and it is not obvious how they should work for a window. For example, would weights apply to rows or to possible start positions, and would replace=True mean replacing rows or replacing windows.

If we go the new methd route, we could define it as positional and iloc-based. Pick a random start in [0, len(obj) - n] and return obj.iloc[start:start+n]. This keeps it independent of index labels or timestamps. If someone wants contiguous in time, they can sort_index() first.

Also, it would be good to agree on behavior when n > length, when n = zero, when the object is empty, and when n is negative. I can add tests for non-RangeIndex, duplicate labels, and MultiIndex as well to confirm the positional behavior stays consistent.

If this direction sounds good, I am happy to work on a PR with implementation, tests, and docs.

[YM2132]

Hey @gautamvarmadatla,

I agree with your point a new method sounds better, adding it to .sample also sort of breaks what .sample() is about (random independent samples).

n > length you'd want an error to be raised right?
n = zero also an error ? as it shouldn't be possible to sample a window of size 0 etc nothing
It'll be worth reviewing how pandas typically handles such cases, etc what if you sample n > len(df)

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[20], line 6
      1 test_df = pd.DataFrame(
      2     np.random.randn(5, 3),
      3     columns=['Column_A', 'Column_B', 'Column_C']
      4 )
----> 6 test_df.sample(10)

File ~/Documents/PyCharm_Projects/gaze_detection/venv313/lib/python3.13/site-packages/pandas/core/generic.py:6140, in NDFrame.sample(self, n, frac, replace, weights, random_state, axis, ignore_index)
   6137 if weights is not None:
   6138     weights = sample.preprocess_weights(self, weights, axis)
-> 6140 sampled_indices = sample.sample(obj_len, size, replace, weights, rs)
   6141 result = self.take(sampled_indices, axis=axis)
   6143 if ignore_index:

File ~/Documents/PyCharm_Projects/gaze_detection/venv313/lib/python3.13/site-packages/pandas/core/sample.py:152, in sample(obj_len, size, replace, weights, random_state)
    149     else:
    150         raise ValueError("Invalid weights: weights sum to zero")
--> 152 return random_state.choice(obj_len, size=size, replace=replace, p=weights).astype(
    153     np.intp, copy=False
    154 )

File numpy/random/mtrand.pyx:1020, in numpy.random.mtrand.RandomState.choice()

ValueError: Cannot take a larger sample than population when 'replace=False'

To your last point, I was hoping to work on this PR myself as it would be my first contribution to pandas. I'd really appreciate your support and I'm sure I'll get stuck, perhaps I give it a crack and let you know how I get on with it?

Finally, what is the typical approach to PRs in pandas? Can we start coding right away or shall we wait for a maintainer to agree on direction?

[gautamvarmadatla]

Hi @YM2132,

sure! Please take a look at this link for contribs. I wanted to be sure with the API design and tests ( including all the edge cases ) with a maintainer before starting.

[YM2132]

@gautamvarmadatla

Sounds good will check it out, however I havent found any reference which says explicitly that we ought to wait for maintainer approval. Suppose it's a slow period too.

Perhaps I can put together an MVP PR and will let you know once I get something up and running, then can start expanding out to edge cases + feedback

[YM2132]

Hey @gautamvarmadatla and maintainers

I've put together a very scrappy first attempt. I used .sample() as the base but have adapted it to sample a single index randomly and then return the next n items including that idx.

It supports random state, but not much else yet. I would like some feedback on the approach and the overall feature from the maintainers before continuing with this feature. I haven't included edge case handling yet, but the idea I think is as .sample() - raise a ValueError is n exceeds length of object.

def sample_window(
            self,
            n: int | None = None,
            axis: Axis | None = None,
            random_state: RandomState | None = None,
    ) -> Self:
        """
        Return a contiguous window of items from the object. The startint point of the window is randomly selected.

        Selects a random index and will return the next `n` items in the axis.

        You can use `random_state` for reproducibility.

        Parameters
        ----------
        n : int, optional
            Number of items from axis to return in the window.
            Default = 1
        random_state : int, array-like, BitGenerator, np.random.RandomState, np.random.Generator, optional
            If int, array-like, or BitGenerator, seed for random number generator.
            If np.random.RandomState or np.random.Generator, use as given.
            Default ``None`` results in sampling with the current state of np.random.
        axis : {0 or 'index', 1 or 'columns', None}, default None
            Axis to sample. Accepts axis number or name. Default is stat axis
            for given data type. For `Series` this parameter is unused and defaults to `None`.

        Returns
        -------
        Series or DataFrame
            A new object of same type as caller containing a contiguous window `n` items
            randomly from the caller object.

        Notes
        -----

        To ensure the window can be sampled we sample the random index from the range [0, len(axis) - n].

        Examples
        --------
        >>> df = pd.DataFrame(
        ...     {
        ...         "num_legs": [2, 4, 8, 0],
        ...         "num_wings": [2, 0, 0, 0],
        ...         "num_specimen_seen": [10, 2, 1, 8],
        ...     },
        ...     index=["falcon", "dog", "spider", "fish"],
        ... )
        >>> df
                num_legs  num_wings  num_specimen_seen
        falcon         2          2                 10
        dog            4          0                  2
        spider         8          0                  1
        fish           0          0                  8

        Sample a contiguous window of size 2. Note that we use `random_state`
        to ensure the reproducibility of the examples.

        >>> df.sample_window(2, random_state=1)
                num_legs  num_wings  num_specimen_seen
        dog            4          0                  2
        spider         8          0                  1

        """
        if axis is None:
            axis = 0

        axis = self._get_axis_number(axis)
        obj_len = self.shape[axis]

        # Process random_state argument
        rs = common.random_state(random_state)

        valid_range = obj_len - n

        print(f"obj_len: {obj_len} n: {n} valid_range: {valid_range}")

        # User defines what size window they want then function will take random sample from which the next N is valid for that size
        # e.g. if n = 10, len(df) == 100, then sample from 0-90
        sampled_index = rs.randint(0, valid_range + 1)  # sample.sample(valid_range, 1, False, None, rs).item()

        # Sampled indices has to be a list of valid indices
        sampled_indices = [sampled_index + i for i in range(n)]

        print(f"Sampled index: {sampled_index} - sampled_indices: {sampled_indices}")

        result = self.take(sampled_indices, axis=axis)

        return result

• I would like to know if the general approach looks correct?
• Any feedback on the code conventions?
• An overall review on the feature and if it fits with the direction that Pandas maintainers want to pursue.

I will add edge cases + tests etc once we have gotten feedback from Pandas team.

[rhshadrach]

Thanks for the request. I am negative on adding this to the already sizable API. It can readily be accomplished in a straightforward and performant manner by users today.

[YM2132]

Hi @rhshadrach, thanks for the comment and for leaving the issue open for discussion.

I was thinking that such a feature would be a logical addition to the already existing .sample() method, for example in time series data a user might want to observe random contiguous blocks of their data or personally I was sorting by a column and wanted to check random contiguous blocks to check the sorting had the intended effect.

Nevertheless I understand your concern with API size, given we have two users here who would like to have this in, do you think a wider study on this use case and if others have brought it up in issues/other forums would help to come to a final decision?

How would you like to proceed with the issue? I'm happy to research the need/present any info you need to make a final decision on whether or not to close the issue.

[rhshadrach]

do you think a wider study on this use case and if others have brought it up in issues/other forums would help to come to a final decision?

I am open to this, but the issue with asking users "would you find this useful" is that it costs nothing to say "yes", but costs pandas a lot in terms of maintenance.

There are hundreds of operations I have no doubt users would find useful, and many variations in behavior on each. It cannot be the goal to provide an API function to call in each case. Rather I think it needs to be the goal of pandas to provide and API where users can implement these themselves in a straightforward and performant manner. It appears to me this is already the case for the operation in question.

[YM2132]

I see your point, to the evidence part I've found a number of SO posts asking about this but most (if not all) do have an answer and using iloc does provide a very simple and neat way to handle this behaviour.

I'll provide some the links I could find here:

• https://stackoverflow.com/questions/67349588/how-to-randomly-extract-n-contiguous-rows-in-a-pandas-dataframe
• https://stackoverflow.com/questions/74199513/pandas-random-n-samples-of-consecutive-rows-pairshttps://stackoverflow.com/questions/59862619/how-can-i-select-a-sequence-of-random-rows-from-a-pandas-dataframe
• https://stackoverflow.com/questions/76095824/pandas-randomly-sample-5-consecutive-rows-based-on-a-condition-value-in-anothe
• https://stackoverflow.com/questions/45938227/random-blocks-of-data-in-pandas

Nevertheless, I imagine there are many features with more questions on the web. Given the existence of iloc and it's nice way to handle the feature I requested in very few LoC we can probably close this unless you think there is more to discuss?

Perhaps, we can add this pattern to some documentation somewhere? I'm not sure of where the best place would be for it but it might be a better way to handle this?