Setup¶
Installation¶
Python 3 requirements:
Installation using pip
pip3 install git+https://git.pyrocko.org/pyrocko/idas-convert
Or installation from source:
git clone https://git.pyrocko.org/pyrocko/idas-convert
cd idas-convert
pyhon3 setup.py install
Check out the help with:
idas_convert -h
Quickstart¶
Before starting the conversion edit the config file. See details of the YAML config in the config.
idas_convert dump_config > my_config.yml
idas_convert my_config.yml
In case of an abort or error the conversion can be resumed with:
idas_convert --resume my_config.yml
Configuration¶
The conversion tools are configured with a YAML file. This file is then _executed_ by idas_convert CLI program.
--- !idas.iDASConvertConfig
# Loading TDMS in parallel and process
nthreads_loading: 1
nthreads_processing: 8
queue_size: 32
processing_batch_size: 8
# Threads used for downsampling the data
nthreads: 8
# Input paths
paths:
- /home/isken/src/idas-convert
# Out path, see pyrocko.io for details
outpath: '%(tmin_year)s%(tmin_month)s%(tmin_day)s/%(network)s.%(station)s_%(tmin_year)s%(tmin_month)s%(tmin_day)s.mseed'
# Overwrite mseed meta information
new_network_code: ID
new_channel_code: HSF
downsample_to: 200.0
# MiniSeed record length
record_length: 4096
# MiniSeed STEIM compression
steim: 2
plugins:
# A plugin handling the communication with the GFZ tage file system
- !idas_convert.gfz_tapes.GFZTapesConfig
enabled: false
bytes_stage: 1T
waterlevel: 0.6
wait_warning_interval: 600.0
release_files: true
path_tapes_mount: /projects/ether/
path_tapes_prefix: /archive_FO1/RAW/
# A Telegram bot to keep up-to-date with the process
- !idas_convert.telegram_bot.TelegramBotConfig
enabled: false
# Telegram API Token
token: 9e98b8c0567149eb861838a1d770be7d
# Telegram Chat ID
chat_id: -1237123123
# A status message will be dispatched every 3600 s
status_interval: 3600.0
Plugins¶
The following plugins can be configured in the plugins list in the YAML file.
Telegram Bot¶
A Telegram bot can be configured to keep up-to-date with the processing progress.
This plugin is forwarding the log levels INFO and WARNING to the chat bot, simply add the bot to a Telegram chat group.
Details about the Telegram token and chat_id can be found here.
- !idas_convert.telegram_bot.TelegramBotConfig
enabled: false
# Telegram API Token
token: 9e98b8c0567149eb861838a1d770be7d
# Telegram Chat ID
chat_id: -1237123123
status_interval: 3600.0
GFZ Tape Interaction¶
The GFZ German Research Centre for Geosciences maintains a tape storage system, details about the system here (in German). This plugin enables seamless inteaction with the SAMFS RPC call procedures, needed to stage and release the files from the hot-storage.
- !idas_convert.gfz_tapes.GFZTapesConfig
enabled: false
bytes_stage: 1T
waterlevel: 0.6
wait_warning_interval: 600.0
release_files: true
path_tapes_mount: /projects/ether/
path_tapes_prefix: /archive_FO1/RAW/