Part 1: GNU Parallel script processing and execution
Why Stack Overflow recommends it
Stack Overflow users link it as the intro video whenever they suggest GNU Parallel in answers about xargs, shell loops, and running commands across many files or cores.
Recommended by 6 people across 19 different months, 2010 to 2021.
Why Unix & Linux Stack Exchange recommends it
Unix & Linux Stack Exchange answerers point people to it as the intro video whenever they suggest GNU Parallel as a replacement for xargs or loops.
Recommended by 3 people across 7 different months, 2010 to 2019.
What it is
A roughly nine minute screencast by GNU Parallel's author, Ole Tange, introducing the tool as a shell utility for running jobs in parallel locally or across remote machines. It shows how GNU Parallel takes lines of input and runs a command on each, as a drop-in alternative to xargs and shell loops.
What people said
Recommends Ole Tange · 2010-06 · Stack Overflow · on “Shell script for testing”
With GNU Parallel [LINK] you can do: /tmp/test > base; seq 1 10 | parallel -k "/tmp/test >output-{}; if diff -q output-{} base; then rm {}; fi" ; echo DONE GNU Parallel is useful for other stuff. Watch the intro video to GNU Parallel: [this video]
Recommends Ole Tange · 2010-07 · Stack Overflow · on “When should xargs be preferred over while-read loops?”
GNU Parallel [LINK] has the advantages from xargs (using -m) and the advantage of while-read with newline as separator and some new features (e.g. grouping of output, parallel running of jobs on remote computers, and context replace). If you have GNU Parallel installed I cannot see a single situation in which you would use xargs . And the only situation in which I would use read-while would be if the block to execute is so big it becomes unreadable to put in a single line (e.g. if it contains if-statements or similar) and you refuse to make a bash function. For all the small scripts I actually find it more readable to use GNU Parallel. paxdiablo's example: echo '1 2 3 4 5' | parallel -m echo
Recommends Ole Tange · 2010-08 · Stack Overflow · on “Bash scripting: Find all filetypes and paths”
If you have GNU Parallel [LINK] installed you can do this: find . -name '*.ext' | parallel echo {} '`dirname {}`' Substitute echo with your favorite bash command and ext with the file extension you are looking for. Watch the intro video for GNU Parallel to learn more: [this video]
Recommends Ole Tange · 2010-08 · Stack Overflow · on “find results piped to zcat and then to head”
If you have GNU Parallel [LINK] installed: find . -name '*.gz' | parallel 'zcat {} | head -n1' Watch the intro video to GNU Parallel at [this video]
Recommends Ole Tange · 2010-08 · Stack Overflow · on “In Unix / Bash, is "xargs -p" a good way to prompt for confirmation before running any command?”
xargs has a nasty tendency to lead to uncomfortable surprises because of the separator problem [LINK] GNU Parallel [LINK] does not have that problem and thus may be a safer choice. seq 10 | parallel -p echo Watch the intro video for GNU Parallel: [this video]
Recommends Ole Tange · 2010-09 · Stack Overflow · on “xargs with multiple arguments”
None of the solutions given so far deals correctly with file names containing space. Some even fail if the file names contain ' or ". If your input files are generated by users, you should be prepared for surprising file names. GNU Parallel [LINK] deals nicely with these file names and gives you (at least) 3 different solutions. If your program takes 3 and only 3 arguments then this will work: (echo a1.txt; echo b1.txt; echo c1.txt; echo a2.txt; echo b2.txt; echo c2.txt;) | parallel -N 3 my-program --file={1} --file={2} --file={3} Or: (echo a1.txt; echo b1.txt; echo c1.txt; echo a2.txt; echo b2.txt; echo c2.txt;) | parallel -X -N 3 my-program --file={} If, however, your program takes as many
Recommends Ole Tange · 2010-09 · Stack Overflow · on “Change sed line separator to NUL to act as "xargs -0" prefilter?”
If none of your file names contain newline, then it may be easier to read a solution using GNU Parallel: filename_listing_command | grep -v '\.py$' | parallel ac Learn more about GNU Parallel [this video]
Recommends Ole Tange · 2010-10 · Stack Overflow · on “Use bash-type `${i%.mp3}` syntax with xargs?”
If you have GNU Parallel installed: find -name '*.mp3' -print0 | parallel -0 mv {} {.}.txt If your file names do not contain \n (newline) this will work too: find -name '*.mp3' | parallel mv {} {.}.txt Watch the intro video to learn more about GNU Parallel [this video]
Recommends Gangadhar · 2010-12 · Stack Overflow · on “Gzip with all cores”
You might want to consider checking GNU parallel [LINK] . I also found this video on youtube [this video] which seems to do what you are looking for.
Recommends Ole Tange · 2010-12 · Unix & Linux Stack Exchange · on “Using xargs with input from a file”
With GNU Parallel you can do: cat urls | parallel curl {} ">" {/} Or: cat persons | parallel curl [LINK] ">" {}.tar Watch the intro video for GNU Parallel to learn more: [this video]