cmemsapi.cmemsapi
#
Main module.
Module Contents#
Functions#
|
Returns working directory where data is saved. |
|
Returns replaced string given string and substitute map. |
|
Returns answer from a yes/no question, read from user's input. |
Returns constraints configuration as |
|
|
Returns Copernicus Marine Credentials. |
|
Check provided Copernicus Marine Credentials are correct. |
Ask the user to input the |
|
|
Convert the |
|
Update a |
|
Convert the |
|
Returns status of binary netCDF file or, if |
|
Returns |
|
Generates as many data requests as required to match initial |
|
Get size from bytes and displays to user in human readable. |
|
Get disk size statistics. |
|
Get size of file(s) in bytes. |
|
Check size of file(s). |
|
Display hard drive statistics to user. |
|
Retrieve a |
|
Retrieve a list of years from a list of netCDF filenames. |
|
Retrieve list of files, based on parameters. |
|
Set output filename based on variables. |
|
Ask user if overwrite action should be performed. |
|
Delete files. |
|
Convert file(s) to one single netCDF-4 file, based on computer limits. |
|
Convert file(s) to one single csv file, based on computer limits. |
|
Convert netcdf file(s) to multiple csv files, based on MS Excel Limits. |
|
Convert file(s) to both netCDF-4 and csv files, based on computer limits. |
|
Post-process the data already located on disk. |
|
Download and post-process files to both compressed and tabular formats, |
|
Method to enable Command Line Interface and to expose only useful method for beginners. |
Attributes#
- cmemsapi.cmemsapi.set_target_directory(local_storage_directory=None)[source]#
Returns working directory where data is saved.
Default value (None) creates a directory (
copernicus-tmp-data
) in the current working directory.- Parameters
local_storage_directory (path or str, optional) – A path object or string. The default is None.
- Returns
target_directory – A path to the directory where data is saved.
- Return type
path
- cmemsapi.cmemsapi.multireplace(tobereplaced, substitute)[source]#
Returns replaced string given string and substitute map.
- cmemsapi.cmemsapi.query(question, default='yes')[source]#
Returns answer from a yes/no question, read from user’s input.
- Parameters
- Raises
ValueError – Raise error to continue asking question until user inputs one of the valid choice.
- Returns
Returns
True
if user validates question,False
otherwise.- Return type
- cmemsapi.cmemsapi.get_config_constraints()[source]#
Returns constraints configuration as
dict
from which data requests will be stacked.- Returns
split_dict – DESCRIPTION.
- Return type
TYPE
- cmemsapi.cmemsapi.get_credentials(file_rc=None, sep='=')[source]#
Returns Copernicus Marine Credentials.
Credentials can be specified in a file or if ommitted, manually by user’s input.
- Parameters
- Raises
SystemExit – Raise an error to exit program at fatal error (wrong credentials etc).
- Returns
copernicus_username (str) – Copernicus Marine username.
copernicus_password (str) – Copernicus Marine password.
- cmemsapi.cmemsapi.check_credentials(user, pwd)[source]#
Check provided Copernicus Marine Credentials are correct.
- Parameters
username (str) – Copernicus Marine Username, provided for free from https://marine.copernicus.eu .
password (str) – Copernicus Marine Password, provided for free from https://marine.copernicus.eu .
- Returns
Returns
True
if credentials are correst,False
otherwise.- Return type
- cmemsapi.cmemsapi.get_viewscript()[source]#
Ask the user to input the
VIEW_SCRIPT
command.- Returns
view_myscript – String representing the
TEMPLATE COMMAND
generated by the webportal. Example is available at https://tiny.cc/get-viewscript-from-web- Return type
- cmemsapi.cmemsapi.viewscript_string_to_dict(viewmy_script)[source]#
Convert the
VIEW SCRIPT
string displayed by the webportal to adict
.- Parameters
viewmy_script (TYPE) – DESCRIPTION.
- Returns
vs_dict – DESCRIPTION.
- Return type
TYPE
- cmemsapi.cmemsapi.get_dates_stack(vs_dict, check_stack, size=None, renew=None)[source]#
Update a
dict
containingVIEW SCRIPT
values with dates for sub-requests.- Parameters
vs_dict (TYPE) – DESCRIPTION.
check_stack (TYPE) – DESCRIPTION.
size (TYPE, optional) – DESCRIPTION. The default is None.
renew (TYPE, optional) – DESCRIPTION. The default is None.
- Returns
vs_dict – DESCRIPTION.
- Return type
TYPE
- cmemsapi.cmemsapi.viewscript_dict_to_string(size=None, strict=None, cmd=None, **kwargs)[source]#
Convert the
dict
containing keys and values of theVIEW SCRIPT
, into a string as displayed by the webportal.- Parameters
size (TYPE, optional) – DESCRIPTION. The default is None.
strict (TYPE, optional) – DESCRIPTION. The default is None.
cmd (TYPE, optional) – DESCRIPTION. The default is None.
**kwargs (TYPE) – DESCRIPTION.
- Returns
command – DESCRIPTION.
- Return type
TYPE
- cmemsapi.cmemsapi.get_data(command=None, user=None, pwd=None, size=None)[source]#
Returns status of binary netCDF file or, if
size
is specified, potential result file size, whose units is kBytes.- Parameters
command (TYPE, optional) – DESCRIPTION. The default is None.
user (TYPE, optional) – DESCRIPTION. The default is None.
pwd (TYPE, optional) – DESCRIPTION. The default is None.
size (TYPE, optional) – DESCRIPTION. The default is None.
- Returns
returncode (TYPE) – DESCRIPTION.
message (TYPE) – DESCRIPTION.
- cmemsapi.cmemsapi.check_data(returncode, message, command=None, user=None, stack=None, size=None)[source]#
Returns
True
if status of the submitted request is successful,False
otherwise.- Parameters
returncode (TYPE) – DESCRIPTION.
message (TYPE) – DESCRIPTION.
command (TYPE, optional) – DESCRIPTION. The default is None.
user (TYPE, optional) – DESCRIPTION. The default is None.
stack (TYPE, optional) – DESCRIPTION. The default is None.
size (TYPE, optional) – DESCRIPTION. The default is None.
- Raises
SystemExit – Raise an error to exit program at fatal error due to server maintenance.
- Returns
valid_check – DESCRIPTION.
- Return type
- cmemsapi.cmemsapi.process_viewscript(target_directory, view_myscript=None, user=None, pwd=None, forcestack=None)[source]#
Generates as many data requests as required to match initial
VIEW_SCRIPT
.- Parameters
- Raises
ValueError – DESCRIPTION.
- Returns
On success, returns path of the output file matching the
VIEW_SCRIPT
data request,False
otherwise.- Return type
TYPE
- cmemsapi.cmemsapi.convert_size_hr(size_in_bytes)[source]#
Get size from bytes and displays to user in human readable.
- Parameters
size_in_bytes (TYPE) – DESCRIPTION.
- Returns
DESCRIPTION.
- Return type
TYPE
- cmemsapi.cmemsapi.get_disk_stat(drive=None)[source]#
Get disk size statistics.
- Parameters
drive (TYPE, optional) – DESCRIPTION. The default is None.
- Returns
disk_stat – DESCRIPTION.
- Return type
TYPE
- cmemsapi.cmemsapi.get_file_size(files)[source]#
Get size of file(s) in bytes.
- Parameters
files (TYPE) – DESCRIPTION.
- Returns
mds_size – DESCRIPTION.
- Return type
TYPE
- cmemsapi.cmemsapi.check_file_size(mds_size, default_nc_size=None)[source]#
Check size of file(s).
- Parameters
mds_size (TYPE) – DESCRIPTION.
default_nc_size (TYPE, optional) – DESCRIPTION. The default is None.
- Returns
check_fs – DESCRIPTION.
- Return type
TYPE
- cmemsapi.cmemsapi.display_disk_stat(mds_size)[source]#
Display hard drive statistics to user.
- Parameters
mds_size (TYPE) – DESCRIPTION.
- Returns
mds_size_hr – DESCRIPTION.
- Return type
TYPE
- cmemsapi.cmemsapi.get_file_pattern(outname, sep='-', rem=-1, advanced=True)[source]#
Retrieve a
file_pattern
from a filename and advanced regex.- Parameters
- Returns
file_pattern – The
file_pattern
extracted fromfilename
.- Return type
- cmemsapi.cmemsapi.get_years(ncfiles, sep='-')[source]#
Retrieve a list of years from a list of netCDF filenames.
- cmemsapi.cmemsapi.get_ncfiles(target_directory, file_pattern=None, year=None)[source]#
Retrieve list of files, based on parameters.
- cmemsapi.cmemsapi.set_outputfile(file_pattern, target_directory, target_out_directory=None, start_year=None, end_year=None)[source]#
Set output filename based on variables.
- Parameters
file_pattern (TYPE) – DESCRIPTION.
target_directory (TYPE) – DESCRIPTION.
target_out_directory (TYPE, optional) – DESCRIPTION. The default is None.
start_year (TYPE, optional) – DESCRIPTION. The default is None.
end_year (TYPE, optional) – DESCRIPTION. The default is None.
- Returns
outputfile – DESCRIPTION.
- Return type
TYPE
- cmemsapi.cmemsapi.over_write(outputfile)[source]#
Ask user if overwrite action should be performed.
- Parameters
outputfile (TYPE) – DESCRIPTION.
- Returns
ow – DESCRIPTION.
- Return type
TYPE
- cmemsapi.cmemsapi.del_ncfiles(ncfiles)[source]#
Delete files.
- Parameters
ncfiles (TYPE) – DESCRIPTION.
- Returns
DESCRIPTION.
- Return type
- cmemsapi.cmemsapi.to_nc4(mds, outputfile)[source]#
Convert file(s) to one single netCDF-4 file, based on computer limits.
- Parameters
mds (TYPE) – DESCRIPTION.
outputfile (TYPE) – DESCRIPTION.
- Returns
nc4 – DESCRIPTION.
- Return type
TYPE
- cmemsapi.cmemsapi.to_csv(mds, outputfile)[source]#
Convert file(s) to one single csv file, based on computer limits.
- Parameters
mds (TYPE) – DESCRIPTION.
outputfile (TYPE) – DESCRIPTION.
- Returns
csv – DESCRIPTION.
- Return type
TYPE
- cmemsapi.cmemsapi.to_mfcsv(input_directory, output_directory, max_depth_level=None)[source]#
Convert netcdf file(s) to multiple csv files, based on MS Excel Limits.
- Parameters
input_directory (TYPE) – DESCRIPTION.
output_directory (TYPE) – DESCRIPTION.
max_depth_level (TYPE, optional) – DESCRIPTION. The default is None.
- Returns
mfcsv – DESCRIPTION.
- Return type
TYPE
- cmemsapi.cmemsapi.to_nc4_csv(ncfiles, outputfile, skip_csv=False, default_nc_size=None)[source]#
Convert file(s) to both netCDF-4 and csv files, based on computer limits.
- Parameters
ncfiles (TYPE) – DESCRIPTION.
outputfile (TYPE) – DESCRIPTION.
skip_csv (TYPE, optional) – DESCRIPTION. The default is False.
default_nc_size (TYPE, optional) – DESCRIPTION. The default is None.
- Returns
nc4 (bool) – DESCRIPTION.
csv (bool) – DESCRIPTION.
check_ow (bool) – DESCRIPTION.
- cmemsapi.cmemsapi.post_processing(outname, target_directory, target_out_directory=None, delete_files=True)[source]#
Post-process the data already located on disk.
Concatenate a complete timerange in a single netCDF-4 file, or if not possible, stack periods on minimum netCDF-4 files (either by year or by month). There is a possibility to delete old files to save space, thanks to convertion from nc3 to nc4 and to convert to CSV, if technically feasible.
- Parameters
outname (TYPE) – DESCRIPTION.
target_directory (TYPE) – DESCRIPTION.
target_out_directory (TYPE, optional) – DESCRIPTION. The default is None.
delete_files (TYPE, optional) – DESCRIPTION. The default is True.
- Raises
SystemExit – DESCRIPTION.
- Returns
processing – DESCRIPTION.
- Return type
See also
get_file_pattern
called from this method
get_ncfiles
called from this method
get_years
called from this method
set_outputfile
called from this method
to_nc4_csv
called from this method
del_ncfiles
called from this method
- cmemsapi.cmemsapi.get(local_storage_directory=None, target_out_directory=None, view_myscript=None, user=None, pwd=None, forcestack=False, delete_files=True)[source]#
Download and post-process files to both compressed and tabular formats, if applicable.
Download as many subsets of dataset required to fulfill an initial data request based on a template command, called
VIEW SCRIPT
generated by Copernicus Marine website (https://marine.copernicus.eu). Then, all files are post-processed locally. e.g to concatenate in a single file, to save space (thanks to nc3 -> nc4), to convert toCSV
(if technically possible), and to delete old files. End-user is guided throughout the process if no parameter is declared. To get started, this function is the main entry point.- Parameters
local_storage_directory (TYPE, optional) – DESCRIPTION. The default is None.
target_out_directory (TYPE, optional) – DESCRIPTION. The default is None.
view_myscript (TYPE, optional) – DESCRIPTION. The default is None.
user (TYPE, optional) – DESCRIPTION. The default is None.
pwd (TYPE, optional) – DESCRIPTION. The default is None.
forcestack (TYPE, optional) – DESCRIPTION. The default is False.
delete_files (TYPE, optional) – DESCRIPTION. The default is True.
- Return type
True.
See also
process_viewscript
Method to parse VIEW SCRIPT
post_processing
Method to convert downloaded data to other format
Examples
Ex 1. Let the user be guided by the script with interactive questions:
>>> cmemstb get
Ex 2. Get data matching a
VIEW SCRIPT
template command passed as parameter:>>> cmemstb get --view_myscript="python -m motuclient --motu https://nrt.cmems-du.eu/motu-web/Motu --service-id GLOBAL_ANALYSIS_FORECAST_PHY_001_024-TDS --product-id global-analysis-forecast-phy-001-024 --longitude-min -20 --longitude-max 45 --latitude-min 25 --latitude-max 72 --date-min \"2019-08-18 12:00:00\" --date-max \"2020-08-31 12:00:00\" --depth-min 0.493 --depth-max 0.4942 --variable thetao --out-dir <OUTPUT_DIRECTORY> --out-name <OUTPUT_FILENAME> --user <USERNAME> --pwd <PASSWORD>"
Notes
For Windows Operating System Users and when using the
--view_myscript
as parameter, you might want to double check thatdouble quote
around dates are well escaped (see above example).