ENH: Improve Filter function with Filter_Columns and Filter_Rows

[speed650]

Feature Type

• Adding new functionality to pandas

• Changing existing functionality in pandas

• Removing existing functionality in pandas

Problem Description

Using the [ ] syntax can get messy/complicated for applying filters to a dataframe. The current Filter() function is also confusing to use. I Propose adding dedicated functions to quickly filter out columns and rows to make Pandas easier to use.

Feature Description

Propose adding 2 new functions

Filter_Columns(), Filter_Rows()

def Filter_Columns( columns: List, inverse:Bool, inplace:Bool )
def Filter_Rows( Rows: List, inverse:Bool, inplace:Bool )

Usage:

Filter_Columns( ['Names', 'Ages' ], inverse=False, inplace=True)

Shows columns for name and age.. Inverse is used to hide show other columns that are not name and age.

def Filter_Rows( [ ('name'==bob), ( 'age' > 20) ] , inverse:Bool, inplace:Bool )

Shows dataframe where the value in names column =bob, and age column >20

Chained

Filter_Columns( ['Names', 'Ages' ], inverse=False, inplace=True).Filter_Rows( [ ('name'==bob), ( 'age' > 20) ] , inverse:Bool, inplace:Bool )

Alternative Solutions

Use [ ] syntex... More confusing

Additional Context

No response

[jbrockmendel]

I’d be on board with deprecating filter so we can change it to a more standard row-filter

[speed650]

That’s good to. Currently it has an input on “items” which is confusing.. you can have a function of “rows” and “columns”, “index”, depending on what you pass it will filter.

…

On Monday, September 25, 2023, jbrockmendel ***@***.***> wrote: I’d be on board with deprecating filter so we can change it to a more standard row-filter — Reply to this email directly, view it on GitHub <#55289 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BCNPCBY5YLYCXD3TK3VLG6DX4ICZ7ANCNFSM6AAAAAA5GSY34Q> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[Cappuchinoo]

take

[phofl]

cc @pandas-dev/pandas-core

[rhshadrach]

Most pandas methods offer an axis argument. I'm not a fan of adding fragmentation to the API by sometimes have axis and sometimes having two methods *_rows and *_columns). I also prefer fewer methods with more arguments as opposed to more methods with fewer arguments, especially where there aren't arguments that go unused in certain cases. I can be on board if we want to move away from axis argument to having two methods - but think it should be across the API rather than just for some methods.

pyspark calls our current method select and uses filter for conditions (similar to what's proposed here). It looks like polars is similar. I like this terminology.

In 2.x, I propose we:

1. Alias filter to select
2. Add filter_cond (or something similar) - in my opinion this does not need, but can have, an axis argument
3. Deprecate filter

In the future, we then have the option to:

4. Alias filter_cond to filter
5. Deprecate filter_cond

For the filter_cond method, it should only be Boolean conditions (e.g. not accept lists of labels), does not need an inverse argument (negation is easy enough), and should have an inplace argument (it can't be done inplace).

[jreback]

@rhshadrach there have been a number of issues to do somewhat similar

not averse but it would likely need a dedicated issue (pls link the original) and comprehensive schedules for this

[Dr-Irv]

I did a PR that didn't go anywhere about having a filter for Index that would create some better syntax for "filtering" on an index, IMHO: #51370 That might cover some of the use cases here.

OP wrote:

The current Filter() function is also confusing to use. I Propose adding dedicated functions to quickly filter out columns and rows to make Pandas easier to use.

I guess the only benefit of this proposal is to allow a list of conditions to be applied.

So why not just add an argument to DataFrame.filter() that allows that list to be specified, and avoid filter_cond ?

[bashtage]

So why not just add an argument to DataFrame.filter() that allows that list to be specified, and avoid filter_cond ?

Most pandas methods offer an axis argument. I'm not a fan of adding fragmentation to the API by sometimes have axis and sometimes having two methods *_rows and *_columns).

I fully agree with these two. DataFrame.filter already accepts axis. It seems the only suggestion here is to add an inverse argument which would make filter drop?

I'm not sure I see where the value add of this proposal lies?

Unless I am missing something, these are pretty easy to do.

def Filter_Columns( columns, inverse=False) -> df.filter(columns)
def Filter_Rows(index, inverse=False) -> df.filter(index, axis=0)
def Filter_Columns( columns, inverse=True) -> df.drop(columns)
def Filter_Rows( columns, inverse=True) -> df.drop(index, axis=0)