Data export
In addition to using the STAC API exposed by the federated catalog, you can get all the data in several formats.
PostgreSQL dump
The PostgreSQL database is exported weekly using pg_dump.
The file can be found at at 🎁 https://api.panoramax.xyz/data/pg_dump/panoramax.pgsql 🎁.
To restore this, you need a running and up to date PostgreSQL database, with Postgis and pg_restore.
Restoring the database will be long and you will need a lot of disk space
The federated catalog contains a lot of data, so the restore may take a while (like several hours) and the resulting database will take several hundreds of GB of disk space.
To get data extract, it might be easier to use the GeoParquet file.
pg_restore --clean --if-exists --no-owner --no-privileges --no-comments --dbname postgres://your_username:your_password@your_localhost:your_port/your_database dump.pgsql
Geoparquet export
The data is also exported weekly in Geoparquet.
The file can be found at 🎁 https://api.panoramax.xyz/data/geoparquet/panoramax.parquet 🎁.
Note
This export is easier to use than the PostgreSQL dump, but it is not as complete as the PostgreSQL dump and the uses will be more limited.
Apache Parquet is a powerful column-oriented data format, built as a modern alternative to CSV files. GeoParquet is an incubating Open Geospatial Consortium (OGC) standard that adds interoperable geospatial types (Point, Line, Polygon) to Parquet.
The federated catalog exports its data following the STAC GeoParquet specification.
Format still in beta
The file format is still in development and should be considered in beta. The specification is also still evolving, so the format may change in the future.
If you need more data or think the format should be different, please open an issue.
File structure
The Geoparquet file follows the STAC GeoParquet specification and has the following fields.
| Field | GeoParquet Type | Details |
|---|---|---|
| stac_extensions | List of Strings | List of used extensions |
| id | String | ID of the item |
| geometry | Binary (WKB) | Point with the position of the picture |
| bbox | Struct of Floats | 2D bounding box of the picture (with xmin, ymin, xmax, ymax fields). Useful to filter data. |
| links | List of Link structs | See Link Struct for more info |
| assets | An Assets struct | See Asset Struct for more info |
| collection | String | The ID of the collection this Item is a part of. Note that, contrary to the main STAC GeoParquet specification usecase, the items in the file will be associated to many collections, and the collection metadata will not be available in the Parquet Metadata. |
| created | TimeStamp | Upload datetime |
| updated | TimeStamp | Last update datetime |
| datetime | TimeStamp | Capture datetime |
| license | String | Associated license |
| original_file:size | Int | Size of the original picture in bytes |
| original_file:name | String | Filename of the original picture |
| panoramax:horizontal_pixel_density | Int | Number of pixels on horizon per field of view degree (see the quality score documentation) |
| quality:horizontal_accuracy | Float | Estimated GPS position precision (in meters) (see the quality score documentation) |
| view:azimuth | Int | Heading of the picture |
| pers:interior_orientation | interior_orientation struct | Information about the camera |
| semantics | Array of semantics struct | Semantic associated to the picture |
| providers | Array of provider struct | List of providers (contributors) |
Structs
Interior orientation
| GeoParquet Type | Details | |
|---|---|---|
| camera_manufacturer | String | Manufacturer of the camera |
| camera_model | String | Model of the camera |
| field_of_view | int | Field of view of the camera |
| focal_length | Float | Focal length of the camera |
| sensor_array_dimensions | Array of int | Dimension of the sensor |
Semantics
The tags in Panoramax are an extra layer of semantics useful to describe the pictures.
| GeoParquet Type | Details | |
|---|---|---|
| key | String | Key |
| value | String | Value |
| on | semantics_association | Detail on where is attached the semantic |
The on field detail the object the semantic is linked to.
If the semantic is linked to:
- the picture, the
onfield is empty, - the whole collection, the
onfield only hasfield=collection, - a part of the picture (an annotation in the Panoramax model), the
onfield will have afield=annotationwith theidof the annotation and itsshape.
The definition of the semantics_association type is:
| GeoParquet Type | Details | |
|---|---|---|
| field | string | Can be collection or annotation (if the semantic is linked to a picture, the whole on field is empty) |
| id | String | Only for annotation, the id of the annotation |
| shape | JSON | Json with the shape of the annotation |
Providers
The STAC Providers are the awesome Panoramax contributors.
| GeoParquet Type | Details | |
|---|---|---|
| id | string | Id of the provider |
| name | string | Name of the provider |
| role | string | Role (will be producer) |
Parquet Metadata
Contrary to the STAC GeoParquet specification, no metadata is stored in the key stac-geoparquet for the moment.
How to use
The Parquet format is great to be remotely queried and it is often not needed to download the whole file.
You can find tutorials on how to use the file in the tutorials section.