cmemsapi package¶
Submodules¶
cmemsapi.cmemsapi module¶
Main module.
-
cmemsapi.cmemsapi.
set_target_directory
(local_storage_directory=None)[source]¶ Returns working directory where data is saved.
Default value (None) creates a directory (
copernicus-tmp-data
) in the current working directory.- Parameters
local_storage_directory (path or str, optional) – A path object or string. The default is None.
- Returns
target_directory – A path to the directory where data is saved.
- Return type
path
-
cmemsapi.cmemsapi.
multireplace
(tobereplaced, substitute)[source]¶ Returns replaced string given string and substitute map.
-
cmemsapi.cmemsapi.
query
(question, default='yes')[source]¶ Returns answer from a yes/no question, read from user’s input.
- Parameters
- Raises
ValueError – Raise error to continue asking question until user inputs one of the valid choice.
- Returns
Returns
True
if user validates question,False
otherwise.- Return type
-
cmemsapi.cmemsapi.
get_config_constraints
()[source]¶ Returns constraints configuration as
dict
from which data requests will be stacked.- Returns
split_dict – DESCRIPTION.
- Return type
TYPE
-
cmemsapi.cmemsapi.
get_credentials
(file_rc=None, sep='=')[source]¶ Returns Copernicus Marine Credentials.
Credentials can be specified in a file or if ommitted, manually by user’s input.
- Parameters
- Raises
SystemExit – Raise an error to exit program at fatal error (wrong credentials etc).
- Returns
copernicus_username (str) – Copernicus Marine username.
copernicus_password (str) – Copernicus Marine password.
-
cmemsapi.cmemsapi.
check_credentials
(user, pwd)[source]¶ Check provided Copernicus Marine Credentials are correct.
- Parameters
username (str) – Copernicus Marine Username, provided for free from https://marine.copernicus.eu .
password (str) – Copernicus Marine Password, provided for free from https://marine.copernicus.eu .
- Returns
Returns
True
if credentials are correst,False
otherwise.- Return type
-
cmemsapi.cmemsapi.
get_viewscript
()[source]¶ Ask the user to input the
VIEW_SCRIPT
command.- Returns
view_myscript – String representing the
TEMPLATE COMMAND
generated by the webportal. Example is available at https://tiny.cc/get-viewscript-from-web- Return type
-
cmemsapi.cmemsapi.
viewscript_string_to_dict
(viewmy_script)[source]¶ Convert the
VIEW SCRIPT
string displayed by the webportal to adict
.- Parameters
viewmy_script (TYPE) – DESCRIPTION.
- Returns
vs_dict – DESCRIPTION.
- Return type
TYPE
-
cmemsapi.cmemsapi.
get_dates_stack
(vs_dict, check_stack, size=None, renew=None)[source]¶ Update a
dict
containingVIEW SCRIPT
values with dates for sub-requests.- Parameters
vs_dict (TYPE) – DESCRIPTION.
check_stack (TYPE) – DESCRIPTION.
size (TYPE, optional) – DESCRIPTION. The default is None.
renew (TYPE, optional) – DESCRIPTION. The default is None.
- Returns
vs_dict – DESCRIPTION.
- Return type
TYPE
-
cmemsapi.cmemsapi.
viewscript_dict_to_string
(size=None, strict=None, cmd=None, **kwargs)[source]¶ Convert the
dict
containing keys and values of theVIEW SCRIPT
, into a string as displayed by the webportal.- Parameters
size (TYPE, optional) – DESCRIPTION. The default is None.
strict (TYPE, optional) – DESCRIPTION. The default is None.
cmd (TYPE, optional) – DESCRIPTION. The default is None.
**kwargs (TYPE) – DESCRIPTION.
- Returns
command – DESCRIPTION.
- Return type
TYPE
-
cmemsapi.cmemsapi.
get_data
(command=None, user=None, pwd=None, size=None)[source]¶ Returns status of binary netCDF file or, if
size
is specified, potential result file size, whose units is kBytes.- Parameters
command (TYPE, optional) – DESCRIPTION. The default is None.
user (TYPE, optional) – DESCRIPTION. The default is None.
pwd (TYPE, optional) – DESCRIPTION. The default is None.
size (TYPE, optional) – DESCRIPTION. The default is None.
- Returns
returncode (TYPE) – DESCRIPTION.
message (TYPE) – DESCRIPTION.
-
cmemsapi.cmemsapi.
check_data
(returncode, message, command=None, user=None, stack=None, size=None)[source]¶ Returns
True
if status of the submitted request is successful,False
otherwise.- Parameters
returncode (TYPE) – DESCRIPTION.
message (TYPE) – DESCRIPTION.
command (TYPE, optional) – DESCRIPTION. The default is None.
user (TYPE, optional) – DESCRIPTION. The default is None.
stack (TYPE, optional) – DESCRIPTION. The default is None.
size (TYPE, optional) – DESCRIPTION. The default is None.
- Raises
SystemExit – Raise an error to exit program at fatal error due to server maintenance.
- Returns
valid_check – DESCRIPTION.
- Return type
-
cmemsapi.cmemsapi.
process_viewscript
(target_directory, view_myscript=None, user=None, pwd=None, forcestack=None)[source]¶ Generates as many data requests as required to match initial
VIEW_SCRIPT
.- Parameters
- Raises
ValueError – DESCRIPTION.
- Returns
On success, returns path of the output file matching the
VIEW_SCRIPT
data request,False
otherwise.- Return type
TYPE
-
cmemsapi.cmemsapi.
convert_size_hr
(size_in_bytes)[source]¶ Get size from bytes and displays to user in human readable.
- Parameters
size_in_bytes (TYPE) – DESCRIPTION.
- Returns
DESCRIPTION.
- Return type
TYPE
-
cmemsapi.cmemsapi.
get_disk_stat
(drive=None)[source]¶ Get disk size statistics.
- Parameters
drive (TYPE, optional) – DESCRIPTION. The default is None.
- Returns
disk_stat – DESCRIPTION.
- Return type
TYPE
-
cmemsapi.cmemsapi.
get_file_size
(files)[source]¶ Get size of file(s) in bytes.
- Parameters
files (TYPE) – DESCRIPTION.
- Returns
mds_size – DESCRIPTION.
- Return type
TYPE
-
cmemsapi.cmemsapi.
check_file_size
(mds_size, default_nc_size=None)[source]¶ Check size of file(s).
- Parameters
mds_size (TYPE) – DESCRIPTION.
default_nc_size (TYPE, optional) – DESCRIPTION. The default is None.
- Returns
check_fs – DESCRIPTION.
- Return type
TYPE
-
cmemsapi.cmemsapi.
display_disk_stat
(mds_size)[source]¶ Display hard drive statistics to user.
- Parameters
mds_size (TYPE) – DESCRIPTION.
- Returns
mds_size_hr – DESCRIPTION.
- Return type
TYPE
-
cmemsapi.cmemsapi.
get_file_pattern
(outname, sep='-', rem=- 1, advanced=True)[source]¶ Retrieve a
file_pattern
from a filename and advanced regex.- Parameters
- Returns
file_pattern – The
file_pattern
extracted fromfilename
.- Return type
-
cmemsapi.cmemsapi.
get_years
(ncfiles, sep='-')[source]¶ Retrieve a list of years from a list of netCDF filenames.
-
cmemsapi.cmemsapi.
get_ncfiles
(target_directory, file_pattern=None, year=None)[source]¶ Retrieve list of files, based on parameters.
-
cmemsapi.cmemsapi.
set_outputfile
(file_pattern, target_directory, target_out_directory=None, start_year=None, end_year=None)[source]¶ Set output filename based on variables.
- Parameters
file_pattern (TYPE) – DESCRIPTION.
target_directory (TYPE) – DESCRIPTION.
target_out_directory (TYPE, optional) – DESCRIPTION. The default is None.
start_year (TYPE, optional) – DESCRIPTION. The default is None.
end_year (TYPE, optional) – DESCRIPTION. The default is None.
- Returns
outputfile – DESCRIPTION.
- Return type
TYPE
-
cmemsapi.cmemsapi.
over_write
(outputfile)[source]¶ Ask user if overwrite action should be performed.
- Parameters
outputfile (TYPE) – DESCRIPTION.
- Returns
ow – DESCRIPTION.
- Return type
TYPE
-
cmemsapi.cmemsapi.
del_ncfiles
(ncfiles)[source]¶ Delete files.
- Parameters
ncfiles (TYPE) – DESCRIPTION.
- Returns
DESCRIPTION.
- Return type
-
cmemsapi.cmemsapi.
to_nc4
(mds, outputfile)[source]¶ Convert file(s) to one single netCDF-4 file, based on computer limits.
- Parameters
mds (TYPE) – DESCRIPTION.
outputfile (TYPE) – DESCRIPTION.
- Returns
nc4 – DESCRIPTION.
- Return type
TYPE
-
cmemsapi.cmemsapi.
to_csv
(mds, outputfile)[source]¶ Convert file(s) to one single csv file, based on computer limits.
- Parameters
mds (TYPE) – DESCRIPTION.
outputfile (TYPE) – DESCRIPTION.
- Returns
csv – DESCRIPTION.
- Return type
TYPE
-
cmemsapi.cmemsapi.
to_mfcsv
(input_directory, output_directory, max_depth_level=None)[source]¶ Convert netcdf file(s) to multiple csv files, based on MS Excel Limits.
- Parameters
input_directory (TYPE) – DESCRIPTION.
output_directory (TYPE) – DESCRIPTION.
max_depth_level (TYPE, optional) – DESCRIPTION. The default is None.
- Returns
mfcsv – DESCRIPTION.
- Return type
TYPE
-
cmemsapi.cmemsapi.
to_nc4_csv
(ncfiles, outputfile, skip_csv=False, default_nc_size=None)[source]¶ Convert file(s) to both netCDF-4 and csv files, based on computer limits.
- Parameters
ncfiles (TYPE) – DESCRIPTION.
outputfile (TYPE) – DESCRIPTION.
skip_csv (TYPE, optional) – DESCRIPTION. The default is False.
default_nc_size (TYPE, optional) – DESCRIPTION. The default is None.
- Returns
nc4 (bool) – DESCRIPTION.
csv (bool) – DESCRIPTION.
check_ow (bool) – DESCRIPTION.
-
cmemsapi.cmemsapi.
post_processing
(outname, target_directory, target_out_directory=None, delete_files=True)[source]¶ Post-process the data already located on disk.
Concatenate a complete timerange in a single netCDF-4 file, or if not possible, stack periods on minimum netCDF-4 files (either by year or by month). There is a possibility to delete old files to save space, thanks to convertion from nc3 to nc4 and to convert to CSV, if technically feasible.
- Parameters
outname (TYPE) – DESCRIPTION.
target_directory (TYPE) – DESCRIPTION.
target_out_directory (TYPE, optional) – DESCRIPTION. The default is None.
delete_files (TYPE, optional) – DESCRIPTION. The default is True.
- Raises
SystemExit – DESCRIPTION.
- Returns
processing – DESCRIPTION.
- Return type
See also
get_file_pattern
called from this method
get_ncfiles
called from this method
get_years
called from this method
set_outputfile
called from this method
to_nc4_csv
called from this method
del_ncfiles
called from this method
-
cmemsapi.cmemsapi.
get
(local_storage_directory=None, target_out_directory=None, view_myscript=None, user=None, pwd=None, forcestack=False, delete_files=True)[source]¶ Download and post-process files to both compressed and tabular formats, if applicable.
Download as many subsets of dataset required to fulfill an initial data request based on a template command, called
VIEW SCRIPT
generated by Copernicus Marine website (https://marine.copernicus.eu). Then, all files are post-processed locally. e.g to concatenate in a single file, to save space (thanks to nc3 -> nc4), to convert toCSV
(if technically possible), and to delete old files. End-user is guided throughout the process if no parameter is declared. To get started, this function is the main entry point.- Parameters
local_storage_directory (TYPE, optional) – DESCRIPTION. The default is None.
target_out_directory (TYPE, optional) – DESCRIPTION. The default is None.
view_myscript (TYPE, optional) – DESCRIPTION. The default is None.
user (TYPE, optional) – DESCRIPTION. The default is None.
pwd (TYPE, optional) – DESCRIPTION. The default is None.
forcestack (TYPE, optional) – DESCRIPTION. The default is False.
delete_files (TYPE, optional) – DESCRIPTION. The default is True.
- Returns
- Return type
True.
See also
process_viewscript
Method to parse VIEW SCRIPT
post_processing
Method to convert downloaded data to other format
Examples
Ex 1. Let the user be guided by the script with interactive questions:
>>> cmemstb get
Ex 2. Get data matching a
VIEW SCRIPT
template command passed as parameter:>>> cmemstb get --view_myscript="python -m motuclient --motu https://nrt.cmems-du.eu/motu-web/Motu --service-id GLOBAL_ANALYSIS_FORECAST_PHY_001_024-TDS --product-id global-analysis-forecast-phy-001-024 --longitude-min -20 --longitude-max 45 --latitude-min 25 --latitude-max 72 --date-min \"2019-08-18 12:00:00\" --date-max \"2020-08-31 12:00:00\" --depth-min 0.493 --depth-max 0.4942 --variable thetao --out-dir <OUTPUT_DIRECTORY> --out-name <OUTPUT_FILENAME> --user <USERNAME> --pwd <PASSWORD>"
Notes
For Windows Operating System Users and when using the
--view_myscript
as parameter, you might want to double check thatdouble quote
around dates are well escaped (see above example).
Module contents¶
Top-level package for Copernicus Marine ToolBox.