Skip to content

Remove filter requirement for data quick search, planet data filter cli command outputs empty filter by default #842

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Feb 23, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
70 changes: 41 additions & 29 deletions docs/cli/cli-data.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,18 +17,18 @@ At this point you should have completed [Step 5](../get-started/quick-start-guid
of the quick start guide, and run your first full data search command:

```
planet data search PSScene filter.json > recent-psscene.json
planet data search PSScene --filter filter.json > recent-psscene.json
```

This saves the latest 100 scenes in a file, that you can open and look at.
This saves the descriptions of the latest 100 standard-quality scenes you have permissions to download in a file, that you can open and look at.

### Pretty printing

You will likely notice that this file is quite wide, with one very long line for each Planet
item returned. You can make for a more readable file by using the `--pretty` flag:

```
planet data search --pretty PSScene filter.json > recent-psscene.json
planet data search --pretty PSScene --filter filter.json > recent-psscene.json
```

The `--pretty` flag is built into most of the CLI calls. But you can also achieve the
Expand All @@ -38,7 +38,7 @@ piping any JSON output through it prints it in a more readable form. So the foll
command will do the same thing as the previous one:

```
planet data search PSScene filter.json | jq > recent-psscene.json
planet data search PSScene --filter filter.json | jq > recent-psscene.json
```

You can read a bit [more about jq]((cli-intro.md#jq) in the CLI intro.
Expand All @@ -49,14 +49,14 @@ You also don't have to save the output to a file. If you don't redirect it into
it will just print out on the console.

```
planet data search PSScene filter.json
planet data search PSScene --filter filter.json
```

If you enter this command you'll see the output stream by. Here you can use jq again, and
it'll often give you nice syntax highlighting in addition to formatting.

```
planet data search PSScene filter.json | jq
planet data search PSScene --filter filter.json | jq
```

### Create filter and search in one call
Expand All @@ -66,14 +66,25 @@ passing the output of the `data filter` command directly to be the input of the
command:

```
planet data filter | planet data search --pretty PSScene -
planet data filter --permission --std-quality | planet data search --pretty PSScene --filter -
```

Note the dash (`-`), which explicitly tells the CLI to use the output from the call that is piped into it.

You can learn more about the pipe command, as well as the `>` command above in the
[Piping & redirection section](cli-intro.md#piping-redirection) of the CLI Introduction.

### Search without filtering

If no filtering is required, the search command can be called directly:

```
planet data search PSScene
```

This outputs the last 100 scenes.


### Search on Item Type

These first searches were done on the [PSScene](https://developers.planet.com/docs/data/psscene/) 'item type', but you
Expand All @@ -82,7 +93,7 @@ its catalog. The item type is the first argument of the `search` command, follow
you can specify any number of item types here:

```
planet data filter | planet data search PSScene,Sentinel2L1C,Landsat8L1G,SkySatCollect -
planet data search PSScene,Sentinel2L1C,Landsat8L1G,SkySatCollect
```

This will search for all the most recent images captured by PlanetScope, SkySat, Sentinel 2 and Landsat 8 satellites.
Expand All @@ -96,7 +107,7 @@ By default the `search` command returns only the 100 first scenes. But with the
under the hood will automatically page through all the results from the API.

```
planet data filter | planet data search --limit 3000 PSScene
planet data search --limit 3000 PSScene
```

Note you can also do a call with no limits if you set the limit to `0`. Though don't use this haphazardly, or you'll be
Expand All @@ -111,13 +122,13 @@ the `planet collect` method to transform the output from the Data API to valid G
output to it:

```console
planet data filter | planet data search PSScene - | planet collect -
planet data search PSScene | planet collect -
```

If you want to visualize this you can save it as a file:

```console
planet data filter | planet data search PSScene - | planet collect - > planet-search.geojson
planet data search PSScene | planet collect - > planet-search.geojson
```

This you can then open with your favorite GIS program, or see this
Expand All @@ -138,13 +149,13 @@ descending order. The options are are:
The lets you do things like get the id of the most recent skysat image taken (that you have download access to):

```console
planet data filter | planet data search SkySatCollect --sort 'acquired desc' --limit 1 -
planet data search SkySatCollect --sort 'acquired desc' --limit 1
```

And you can also just get the ID, using `jq`

```console
planet data filter | planet data search SkySatCollect --sort 'acquired desc' --limit 1 - | jq -r .id
planet data search SkySatCollect --sort 'acquired desc' --limit 1 - | jq -r .id
```


Expand Down Expand Up @@ -202,14 +213,14 @@ of Iowa. You can copy it and save as a file called `geometry.geojson`
And then run it with this command:

```console
planet data filter --geom geometry.geojson | planet data search PSScene -
planet data filter --geom geometry.geojson | planet data search PSScene --filter -
```

Note that by default all searches with the command-line return 100 results, but you can easily increase that with
the `--limit` flag:

```console
planet data filter --geom geometry.geojson | planet data search --limit 500 PSScene -
planet data filter --geom geometry.geojson | planet data search --limit 500 PSScene --filter -
```

Creating geometries for search can be annoying in a command-line workflow, but there are some ideas in the
Expand All @@ -220,7 +231,7 @@ Creating geometries for search can be annoying in a command-line workflow, but t
Some of the most common filtering is by date. You could get all imagery acquired before August 2021:

```console
planet data filter --date-range acquired lt 2021-08-01 | planet data search PSScene -
planet data filter --date-range acquired lt 2021-08-01 | planet data search PSScene --filter -
```

The 'operator' in this case is 'less than' (`lt`). The options are:
Expand All @@ -236,7 +247,7 @@ do a search for all images in July of 2021:

```console
planet data filter --date-range acquired gte 2021-07-01 --date-range acquired lt 2021-08-01 | \
planet data search PSScene -
planet data search PSScene --filter -
```

The date input understands [RFC 3339](https://datatracker.ietf.org/doc/html/rfc3339) and
Expand All @@ -246,7 +257,7 @@ on July 1st 2021:

```console
planet data filter --date-range acquired gte 2021-07-01:06:20:10 --date-range acquired lt 2021-07-01:06:20:15 | \
planet data search PSScene -
planet data search PSScene --filter -
```

### Range Filter
Expand All @@ -255,7 +266,7 @@ The range filter uses the same operators as the date filter, but works against a
of these tend to be ones about cloudy pixels. For example you can search for data with clear pixels greater than 90%:

```console
planet data filter --range clear_percent gt 90
planet data filter --range clear_percent gt 90 | planet data search PSScene --filter -
```

### String-In Filter
Expand All @@ -264,19 +275,20 @@ For properties that are strings you can use the `string-in` filter. For example
with PS2 instrument:

```console
planet data filter --string-in instrument PS2 | planet data search PSScene -
planet data filter --string-in instrument PS2 | planet data search PSScene --filter -
```

You can specify multiple strings to match, with a comma:

```console
planet data filter --string-in instrument PS2,PSB.SD | planet data search PSScene -
planet data filter --string-in instrument PS2,PSB.SD | planet data search PSScene --filter -

```

Another example is to select all data in a single strip:

```console
planet data filter --string-in strip_id 5743640 | planet data search PSScene -
planet data filter --string-in strip_id 5743640 | planet data search PSScene --filter -
```

Note that in all these commands we are piping the results into the search. If you don't include the pipe then you'll
Expand All @@ -287,13 +299,13 @@ get the filter output, which can be interested to inspect to see exactly what is
You can limit your search to only data with a particular asset, for example search just for 8-band analytic assets:

```console
planet data filter --asset ortho_analytic_8b_sr | planet data search PSScene -
planet data filter --asset ortho_analytic_8b_sr | planet data search PSScene --filter -
```

Or 8-band assets that also have a UDM.

```console
planet data filter --asset ortho_analytic_8b_sr --asset udm2 | planet data search PSScene -
planet data filter --asset ortho_analytic_8b_sr --asset udm2 | planet data search PSScene --filter -
```

You can find the list of available assets in each Item Type Page, like
Expand All @@ -306,14 +318,14 @@ sure you got the asset right, and it's valid for the item-types you're searching

### Permission Filter

The 'permission filter' is set to true by default, since most people want to search only for data they have access to
and are able to download. But if you'd like to just get search Planet's catalog and get a sense of what is out there
you can set the permission filter to false:
By default, no search filters are applied. However, many people want to search only for data they have access to download
that are of standard (aka not test) quality. Therefore, these filters can be easily added with the `--permission` and
`--std-quality` flags. To use the permission and standard quality filters:

```console
planet data filter --permission false --asset ortho_analytic_8b_sr | planet data search PSScene -
planet data filter --permission --std-quality --asset ortho_analytic_8b_sr | planet data search PSScene --filter -
```

## Stats

TODO
TODO
7 changes: 3 additions & 4 deletions docs/cli/cli-intro.md
Original file line number Diff line number Diff line change
Expand Up @@ -84,14 +84,13 @@ one command to be the input for the next one. So instead of having to save to a
then referring to it you can just do it all in one call:

```
planet data filter --range cloud_percent lt 10 | planet data search-quick PSScene -
planet data filter --range cloud_percent lt 10 | planet data search PSScene --filter -
```

The pipe says to take the output of the first command and pass it to the input of
the second. You'll notice that the planet command has a dash (`-`), this is a convention
that is often used by different CLI programs to explicitly say 'read from
standard out'. Most Planet CLI commands require it, but one or two will implicitly
read from standard out if it's not explicitly included. Using the dash to mean
standard out'. Using the dash to mean
'read from standard out' is a general convention used by many programs, but it's
not universal, so check the docs of the program you're using as to how it reads
from piped input. For example GDAL/OGR uses a specific `/vsistdin/` convention to
Expand Down Expand Up @@ -174,7 +173,7 @@ represent GeoJSON features, the JSON blob is a GeoJSON FeatureCollection.
Otherwise, the JSON blob is a list of the individual results.

```console
$ planet data filter | planet data search PSScene - | planet collect -
$ planet data search PSScene | planet collect -
```

This gives you a fully compliant GeoJSON FeatureCollection, which is
Expand Down
31 changes: 14 additions & 17 deletions docs/cli/cli-tips-tricks.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ JSON from the text box:
On a mac you can use `pbpaste` to grab whatever is currently in your clipboard:

```console
pbpaste | planet data filter --geom - | planet data search SkySatCollect -
pbpaste | planet data filter --geom - | planet data search SkySatCollect --filter -
```

(TODO: Find alternatives for windows and linux)
Expand Down Expand Up @@ -94,7 +94,7 @@ Or also on Placemark, which tends to perform a bit better (especially when you g
For both it's recommended to pass the output through `planet collect` to get properly formatted GeoJSON:

```console
planet data filter --string-in strip_id 5743669 | planet data search PSScene - | planet collect - | pbcopy
planet data filter --string-in strip_id 5743669 | planet data search PSScene --filter - | planet collect - | pbcopy
```

(TODO: Get pbcopy equivalents for windows and linux)
Expand All @@ -108,15 +108,15 @@ The following command will get the latest SkySat image captured, upload to githu
your browser to see it:

```console
planet data filter | planet data search SkySatCollect - --sort 'acquired desc' --limit 1 \
planet data search SkySatCollect --sort 'acquired desc' --limit 1 \
| planet collect - | jq | gh gist create -f latest-skysat.geojson -w
```

Or you can show all ps-scenes in a strip on github gist.
(You may need to reload the page, for some reason it doesn't always showing up immediately after open)

```console
planet data filter --string-in strip_id 5743640 | planet data search PSScene - \
planet data filter --string-in strip_id 5743640 | planet data search PSScene --filter - \
| planet collect - | gh gist create -f ps-search.geojson -w
```

Expand All @@ -125,9 +125,9 @@ TODO: get a command that gets the latest strip id and uses that as input in one
This current command doesn't quite work.

```console
strip-id=`planet data filter | planet data search PSScene - --limit 1 \
strip-id=`planet data search PSScene --limit 1 \
| jq -r '.properties.strip_id' | sed 's/\\[tn]//g'`
planet data filter --string-in strip_id $stripid | planet data search PSScene -
planet data filter --string-in strip_id $stripid | planet data search PSScene --filter -
```


Expand All @@ -146,23 +146,20 @@ Once it's set up you can just pipe any search command directly to `kepler` (it u

```console
curl -s https://storage.googleapis.com/open-geodata/ch/vermont.json \
| planet data filter --permission false --geom - \
| planet data search PSScene - \
| planet data filter --geom - \
| planet data search PSScene --filter - \
| kepler
```

Note `--permission false` is added to these examples so that anyone with Planet access can try them, even if you don't
have download permission for the area shown.

(TODO: Add animated gif, showing some options)

Kepler really excels at larger amounts of data, so try it out with larger limits:

```console
curl -s https://storage.googleapis.com/open-geodata/ch/vermont.json \
| planet data filter --permission false --geom - \
| planet data filter --geom - \
| planet data search PSScene,Sentinel2L1C,Landsat8L1G,SkySatCollect,Sentinel1 \
--sort 'acquired desc' --limit 1500 - \
--sort 'acquired desc' --limit 1500 --filter - \
| kepler
```

Expand All @@ -176,8 +173,8 @@ And you can bring it all together using Placemark for input and Kepler for outpu

```console
curl -s https://api.placemark.io/api/v1/map/a0BWUEErqU9A1EDHZWHez/feature/91a07390-0652-11ed-8fdd-15633e4f8f01 \
| planet data filter --permission false --geom - \
| planet data search PSScene,Landsat8L1G,SkySatCollect,Sentinel1 - | kepler
| planet data filter --geom - \
| planet data search PSScene,Landsat8L1G,SkySatCollect,Sentinel1 --filter - | kepler
```

#### Large Dataset Visualization
Expand All @@ -188,7 +185,7 @@ to disk. Getting 20,000 skysat collects will take at least a couple of minutes,
100 megabytes of GeoJSON on disk.

```console
planet data filter | planet data search SkySatCollect --limit 20000 > skysat-large.geojson
planet data search SkySatCollect --limit 20000 > skysat-large.geojson
```

Kepler can fairly easily handle 20,000 skysat footprints, try:
Expand Down Expand Up @@ -274,7 +271,7 @@ mapshaper -i footprints.geojson -simplify 15% -o simplified.geojson
Once you find a simplification amount you're happy with you can use it as a piped output.

```console
planet data filter | planet data search --limit 20 SkySatCollect - | planet collect - | mapshaper -i - -simplify 15% -o skysat-ms2.geojson
planet data search --limit 20 SkySatCollect - | planet collect - | mapshaper -i - -simplify 15% -o skysat-ms2.geojson
```

Mapshaper also has more simplification algorithms to try out, so we recommend diving into the
Expand Down
8 changes: 4 additions & 4 deletions docs/get-started/quick-start-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,10 +100,10 @@ In this step, you search for the most recent PSScene images available to downloa
One of the commands you’ll use most frequently is `planet data filter`. This “convenience method” creates the JSON you need to run other commands. Run it with no arguments to see how it works by default:

```console
planet data filter
planet data filter --permission --std-quality
```

Look at the console output to see some default filters. `PermissionFilter` filters the output to only contain imagery that you have permission to download. You’ll also see `quality_category`, which means the output lists only images in the [`standard quality` category](https://developers.planet.com/docs/data/planetscope/#image-quality-standard-vs-test-imagery).
Look at the console output to see some default filters. `PermissionFilter` filters the output to only contain imagery that you have permission to download. You’ll also see `quality_category`, which means the output lists only images in the [`standard quality` category](https://developers.planet.com/docs/data/planetscope/#image-quality-standard-vs-test-imagery). Without these options, an empty filter is generated which would be used to disable filtering and simply return all results.

!!!note "The --help switch is your friend"
You can do a lot with this `filter` command. We recommend running `planet data filter --help` often to get a reference of how the commands work.
Expand All @@ -113,13 +113,13 @@ Look at the console output to see some default filters. `PermissionFilter` filte
Run the filter command and save it to a file named `filter.json`:

```console
planet data filter > filter.json
planet data filter --permission --std-quality > filter.json
```

Then use that file with the search command and save the results to another file named `recent-psscene.json`.

```console
planet data search PSScene filter.json > recent-psscene.json
planet data search PSScene --filter filter.json > recent-psscene.json
```

Open `recent-psscene.json` to see the 100 most recent PSScene images you have permissions to actually download.
Expand Down
Loading