helpTextVerbose
immutable
auto helpTextVerbose =
q"EOS
Synopsis: tsv-split [options] [file...]
Split input lines into multiple output files. There are three modes of
operation:
* Fixed number of lines per file (--l|lines-per-file NUM): Each input
block of NUM lines is written to a new file. Similar to Unix 'split'.
* Random assignment (--n|num-files NUM): Each input line is written to a
randomly selected output file. Random selection is from NUM files.
* Random assignment by key (--n|num-files NUM, --k|key-fields FIELDS):
Input lines are written to output files using fields as a key. Each
unique key is randomly assigned to one of NUM output files. All lines
with the same key are written to the same file.
Output files: By default, files are written to the current directory and
have names of the form 'part_NNN<suffix>', with 'NNN' being a number and
<suffix> being the extension of the first input file. If the input file is
'file.txt', the names will take the form 'part_NNN.txt'. The suffix is
empty when reading from standard input. The numeric part defaults to 3
digits for '--l|lines-per-files'. For '--n|num-files' enough digits are
used so all filenames are the same length. The output directory and file
names are customizable.
Header lines: There are two ways to handle input with headers: write a
header to all output files (--H|header), or exclude headers from all
output files ('--I|header-in-only'). The best choice depends on the
follow-up processing. All tsv-utils tools support header lines in multiple
input files, but many other tools do not. For example, GNU parallel works
best on files without header lines.
Random assignment (--n|num-files): Random distribution of records to a set
of files is a common task. When data fits in memory the preferred approach
is usually to shuffle the data and split it into fixed sized blocks. E.g.
'tsv-sample data.tsv | tsv-split -l NUM'. However, alternate approaches
are needed when data is too large for convenient shuffling. tsv-split's
random assignment feature is useful in this case. Each input line is
written to a randomly selected output file. Note that output files will
have similar but not identical numbers of records.
Random assignment by key (--n|num-files NUM, --k|key-fields FIELDS): This
splits a data set into multiple files sharded by key. All lines with the
same key are written to the same file. This partitioning enables parallel
computation based on the key. For example, statistical calculation
('tsv-summarize --group-by') or duplicate removal ('tsv-uniq --fields').
These operations can be parallelized using tools like GNU parallel, which
simplifies concurrent operations on multiple files. Fields are specified
using field number or field name. Field names require that the input file
has a header line. Use '--help-fields' for details about field names.
Random seed: By default, each tsv-split invocation using random assignment
or random assignment by key produces different assignments to the output
files. Using '--s|static-seed' changes this so multiple runs produce the
same assignments. This works by using the same random seed each run. The
seed can be specified using '--v|seed-value'.
Appending to existing files: By default, an error is triggered if an
output file already exists. '--a|append' changes this so that lines are
appended to existing files. (Header lines are not appended to files with
data.) This is useful when adding new data to files created by a previous
tsv-split run. Random assignment should use the same '--n|num-files' value
each run, but different random seeds (avoid '--s|static-seed'). Random
assignment by key should use the same '--n|num-files', '--k|key-fields',
and seed ('--s|static-seed' or '--v|seed-value') each run.
Max number of open files: Random assignment and random assignment by key
are dramatically faster when all output files are kept open. However,
keeping a large numbers of open files can bump into system limits or limit
resources available to other processes. By default, tsv-split uses up to
4096 open files or the system per-process limit, whichever is smaller.
This can be changed using '--max-open-files', though it cannot be set
larger than the system limit. The system limit varies considerably between
systems. On many systems it is unlimited. On MacOS it is often set to 256.
Use Unix 'ulimit' to display and modify the limits:
* 'ulimit -n' - Show the "soft limit". The per-process maximum.
* 'ulimit -Hn' - Show the "hard limit". The max allowed soft limit.
* 'ulimit -Sn NUM' - Change the "soft limit" to NUM.
Examples:
# Split a 10 million line file into 1000 files, 10,000 lines each.
# Output files are part_000.txt, part_001.txt, ... part_999.txt.
tsv-split data.txt --lines-per-file 10000
# Same as the previous example, but write files to a subdirectory.
tsv-split data.txt --dir split_files --lines-per-file 10000
# Split a file into 10,000 line files, writing a header line to each
tsv-split data.txt -H --lines-per-file 10000
# Same as the previous example, but dropping the header line.
tsv-split data.txt -I --lines-per-file 10000
# Randomly assign lines to 1000 files
tsv-split data.txt --num-files 1000
# Randomly assign lines to 1000 files while keeping unique entries
# from the 'url' field together.
tsv-split data.tsv -H -k url --num-files 1000
# Randomly assign lines to 1000 files. Later, randomly assign lines
# from a second data file to the same output files.
tsv-split data1.tsv -n 1000
tsv-split data2.tsv -n 1000 --append
# Randomly assign lines to 1000 files using field 3 as a key.
# Later, add a second file to the same output files.
tsv-split data1.tsv -n 1000 -k 3 --static-seed
tsv-split data2.tsv -n 1000 -k 3 --static-seed --append
# Change the system per-process open file limit for one command.
# The parens create a sub-shell. The current shell is not changed.
( ulimit -Sn 1000 && tsv-split --num-files 1000 data.txt )
Options:
EOS";
tsv_utils tsv_split
functionsstatic variablesstructsvariables