===== 2. Have a Good Command ===== ==== Concatenating with cat ==== * General way to read content with cat:cat file1 file2 file3 ... * Combine stdin with a file:echo 'Text through stdin' | cat - file.txt * Removing extra (more than 2 consecutive) blank lines:cat -s file * Other cat flags:# Display tabs as ^I: cat -T file.py # Display line numbers: cat -n file.txt ==== Recording and playing back of terminal sessions ==== * We can start recording a session with:script -t 2> timing.log -a output.session ... # type commands here ... exit * Replay the commands with:scriptreplay timing.log output.session ==== Finding files and file listing ==== * Find all the files in current directory:find base_path # For instance: find . -print # We can use -print0 to use '\0' as delimiting character. # This is usefull when flename contains spaces * Search base on filename or regular expression:find /home/slynux -name "*.txt" -print # Using the option iname to ignore case: find . -iname "example*" -print # or condition for multiple criteria: find . \( -name "*.txt" -o -name "*.pdf" \) -print # Using path argument: find /home/users -path "*/slynux/*" -print # Using regex argumen to match paths based on regular expressions: find . -regex ".*\(\.py\|\.sh\)$" # or iregex to ignore case: find . -iregex ".*\(\.py\|\.sh\)$" * Negating arguments:# Exclude things that match a pattern: find . ! -name "*.txt" -print * Search based on directory depth:# Only printing files in the current directory: find . -maxdepth 1 -name "f*" -print # Or using mindepth: find . -mindepth 2 -name "f*" -print # note that these flags should be specified as third argument for find to improve efficiency. * Search based on file type: find . -type d -print # find directories find . -type f -print # find regular files find . -type l -print # find symlinks * Search on file times: # we can use the flags: -atime : access time -mtime : modification time -ctime : change time # The provided inter value is the number of days: find . -type f -atime -7 -print # all files accessed within the last 7 days find . -type f -atime 7 -print # all files accessed exactly 7 days ago. find . -type f -atime +7 -print # all files that were accessed more that 7 days ago. # we can also used minutes based flags: -amin -mmin -cmin # We can also find files newer that a given file: find . -type f -newer file.txt -print * Search based on file size:find . -type f -size +2k # files bigger than 2KB find . -type f -size -2k # smaller than 2kB # instead of 'k', we can use 'M', 'G' * Deleting file matches:find . -type f -name "*.swp" -delete * Match based on file permissions:find . -type f -perm 644 -print # For instance: find . -type f -name "*.php" ! -perm 644 -print # Or search based on user: find . -type f -user slynux -print * Executing commands with find:find . -type f -user root -exec chown slynux {} \; # In the previous command: # '{}' will be replaced by each filename. # if we want to run a command with a list of files as parameters then # we just replace ';' with '+' * To concatenate multiple files for instance:find . -type f -name "*.c" -exec cat {} \;>all_c_files.txt * To copy all the .txt files that are older than 10 days to a directory OLD: find . -type f -mtime +10 -name "*.txt" -exec cp {} OLD \; * If we need multiple commands with exec then we have to write a script file. * Combine exec with printf:find . -type f -name "*.txt" -exec printf "Text file: %s\n" {} \; * Skipping specified directories in find:find devel/source_path \( -name ".git" -prune \) -o \( -type f -print \) ==== Playing with xargs ==== * Converting multiple lines to a single line output:cat example.txt | xargs * Converting single-line into multiple-line output:cat example.txt | xargs -n 3 * Specify delimiter:echo "splitXsplitXsplitXsplit" | xargs -d X * Provide one/more arguments from a file listing to a command:cat args.txt | xargs -n 1 ./cecho.sh # To provide n arguments we use the prototype: INPUT | xargs -n X # to provide all the arguments at once: cat args.txt | xargs ./ccat.sh * We can specify the -I flag to provide a replacement string (only with one argument per command execution):cat args.txt | xargs -I {} ./cecho.sh -p {} -l * Using xargs with find:find . -type f -name "*.txt" -print | xargs rm -f # Safer implementation is: find . -type f -name "*.txt" -print0 | xargs -0 rm -f * Count number of lines of C code:find source_code_dir_path -type f -name "*.c" -print0 | xargs -0 wc -l # One could also consider using the SLOCCount utility * Using a subshell script instead of xargs: cat files.txt | xargs -I {} cat {} # is equivalent to: cat files.txt | ( while read arg; do cat $arg; done ) ==== Translating with tr ==== * Simple translation:echo "HELLO WHO IS THIS" | tr 'A-Z' 'a-z' * Other encryptions:echo 12345 | tr '0-9' '9876543210' # encrypt echo 87654 | tr '9876543210' '0-9' # decrypt # ROT13 encryption: echo "tr came, tr saw, tr conquered." | tr 'a-zA-Z' 'n-za-mN-ZA-M' # decryption: echo ge pnzr, ge fnj, ge pbadhrerq. | tr 'a-zA-Z' 'n-za-mN-ZA-M' * Converting tab to space:tr '\t' ' ' < file.txt * Deleting characters:echo "Hello 123 world 456" | tr -d '0-9' * Complementing character set:echo hello 1 char 2 next 4 | tr -d -c '0-9 \n' * Squeezing characters with tr:echo "GNU is not UNIX. Recursive right ?" | tr -s ' ' * Compute a sum from a file:# Assuming sum.txt contains one number per line: cat sum.txt | echo $[ $(tr '\n' '+' ) 0 ] * Can be used with character sets like: alnum, alpha, cntrl, digit, graph, lower, print, punct, space, upper, xdigit: tr [:class:] [:class:] ==== Checksum and verification ==== * To compute the checksum we can use:$ md5sum filename 68b329da9893e34099c7d8ad5cb9c940 filename # We can redirect the output to file: $ md5sum filename > file_sum.md5 # Prototype is: $ md5sum file1 file2 file3 ... # This will output one line per file. * To verify the integrity of a file:$ md5sum -c file_sum.md5 # This will output a message whether checksum matches or not. # Alternatively: $ md5sum -c *.md5 * Usage of SHA-1 is similar: replace **md5sum** with **sha1sum**. * We can compute checksum for directory with md5deep and sha2deep:$ md5deep -rl directory_path > directory.md5 # -r to enable recursive traversal # -l for using relative path. By default it writes absolute file path in output # Alternatively we can use find: $ find directory_path -type f -print0 | xargs -0 md5sum >> directory.md5 ==== Cryptographic tools and hashes ==== * Encryption with crypt:$ crypt output_file Enter passphrase: # alternatively, we can provide the passphrase on the command line: $ crypt PASSPHRASE encrypted_file # to decrypt: $ crypt PASSPHRASE -d output_file * Encryption with gpg:$ gpg -c filename # to decrypt: $ gpg filename.gpg * Encryption with base64:$ base64 filename > outputfile # or: $ cat file | base64 > outputfile # To decode: $ base64 -d file > outputfile # or: $ cat base64_file | base64 -d > outputfile * md5sum and sha1sum can also be used to store passwords for instance (but bcrypt and sha512sum are recommended instead) * Generate shadow password with openssl:$ opensslpasswd -1 -salt SALT_STRING PASSWORD $1$SALT_STRING$323VkWkSLHuhbt1zkSsUG. ==== Sorting unique and duplicates ==== * Sort a given set of files:$ sort file1.txt file2.txt > sorted.txt # or: $ sort file1.txt file2.txt -o sorted.txt # For numerical sorting: $ sort -n file.txt # To sort in reverse order: $ sort -r file.txt # To sort by month: $ sort -M months.txt # To merge 2 sorted files: $ sort -m sorted1 sorted2 # To find unique lines in sorted file: $ sort file1.txt file2.txt | uniq * To check if a file is already sorted we check the result of sort:#!/bin/bash #Desc: Sort sort -C filename ; if [ $? -eq 0 ]; then echo Sorted; else echo Unsorted; fi * Sort by a column in text file:$ cat data.txt 1 mac 2000 2 winxp 4000 3 bsd 1000 4 linux 1000 # we use the -k flag to specify the column to use: # Sort reverse by column1 $ sort -nrk 1 data.txt 4 linux 1000 3 bsd 1000 2 winxp 4000 1 mac 2000 # -nr means numeric and reverse # Sort by column 2 $ sort -k 2 data.txt 3 bsd 1000 4 linux 1000 1 mac 2000 2 winxp 4000 * Specify a range for the key:$ cat data.txt 1010hellothis 2189ababbba 7464dfddfdfd $ sort -nk 2,3 data.txt # To use first character as key: $ sort -nk 1,1 data.txt # To use a \0 separator: $ sort -z data.txt | xargs -0 #Zero terminator is used to make safe use with xargs # To ignore leading blank and use dictionnary order: $ sort -bd unsorted.txt * Usage of uniq:$ sort unsorted.txt | uniq * Display only the unique lines:$ uniq -u sorted.txt * Count how many times each line appears:$ sort unsorted.txt | uniq -c 1 bash 1 foss 2 hack * Find the duplicate lines:$ sort unsorted.txt | uniq -d hack * Specify start and width for uniqueness computation:$ cat data.txt u:01:gnu d:04:linux u:01:bash u:01:hack $ sort data.txt | uniq -s 2 -w 2 d:04:linux u:01:bash * Terminate lines with \0 separator:$ uniq -z file.txt ==== Temporary file naming and random numbers ==== * Create a temporary file:$ filename=`mktemp` $ echo $filename /tmp/tmp.8xvhkjF5fH * Create temporary directory:$ dirname=`mktemp -d` $ echo $dirname tmp.NI8xzW7VRX * To just generate a filename without actually creating it:$ tmpfile=`mktemp -u` $ echo $tmpfile /tmp/tmp.RsGmilRpcT * Create temp file according to template:$mktemp test.XXX test.2tc ==== Splitting files and data ==== * Splitting a file:$ split -b 10k data.file $ ls data.file xaa xab xac xad xae xaf xag xah xai xaj * To use numeric suffixes:$ split -b 10k data.file -d -a 4 * Specify a filename prefix:$ split -b 10k data.file -d -a 4 split_file * To split based on number of lines:$ split -l 10 data.file * csplit can be used to split based on file content: csplit server.log /SERVER/ -n 2 -s {*} -f server -b "%02d.log" ; rm server00.log ==== Slicing filenames based on extension ==== * Extracting the name from **name.extension**:file_jpg="sample.jpg" name=${file_jpg%.*} echo File name is: $name * Extracting the extension from **name.extension**:extension=${file_jpg#*.} * Note the the oerator % is non-greedy (eg. finds the minimal match). Instead operator % % is greedy:$ VAR=hack.fun.book.txt $ echo ${VAR%.*} hack.fun.book $ echo ${VAR%%.*} hack * We also have operator ## similar to # but greedy:$ VAR=hack.fun.book.txt $ echo ${VAR#*.} fun.book.txt $ echo ${VAR##*.} txt ==== Renaming and moving files in bulk ==== * Rename all image files in the current directory:#!/bin/bash #Filename: rename.sh #Desc: Rename jpg and png files count=1; for img in `find . -iname '*.png' -o -iname '*.jpg' -type f -maxdepth 1` do new=image-$count.${img##*.} echo "Renaming $img to $new" mv "$img" "$new" let count++ done * Renaming *.JPG to *.jpg:$ rename *.JPG *.jpg * Replace spaces with underscore:$ rename 's/ /_/g' * * Convert from uppr to lower or opposite:$ rename 'y/A-Z/a-z/' * $ rename 'y/a-z/A-Z/' * * Move all mp3 in a folder:$ find path -type f -name "*.mp3" -exec mv {} target_dir \; * Recursive rename:$ find path -type f -exec rename 's/ /_/g' {} \; ==== Spell checking and dictionary manipulation ==== * Dictionary files found in /usr/share/dict/ * Check if word is part of dictionary:#!/bin/bash #Filename: checkword.sh word=$1 grep "^$1$" /usr/share/dict/british-english -q if [ $? -eq 0 ]; then echo $word is a dictionary word; else echo $word is not a dictionary word; fi # Usage as: $ ./checkword.sh ful ful is not a dictionary word $ ./checkword.sh fool fool is a dictionary word * or we can use **aspell**. * List all words in a file starting with a given word as follows: $ look word filepath # or: $ grep "^word" filepath ==== Automating interactive input ==== * Automate an input for a command with:$ echo -e "1\nhello\n" | ./interactive.sh You have entered 1, hello # -e flag for echo means 'interpret escape sequences' * The **expect** program can be used when the input order is not always the same. ==== Making commands quicker by running parallel processes ==== * Run multiple instances of scripts with for instance:#/bin/bash #filename: generate_checksums.sh PIDARRAY=() for file in File1.iso File2.iso do md5sum $file & PIDARRAY+=("$!") # $! : retrieves the PID of the last background process. done wait ${PIDARRAY[@]}