public:books:linux_shell_scripting_cookbook:chapter_3

  • loopback files: contain a filesystem itself and can be mounted.
  • Create a file with dd:
    $ dd if=/dev/zero of=junk.data bs=1M count=1
    1+0 records in
    1+0 records out
    1048576 bytes (1.0 MB) copied, 0.00767266 s, 137 MB/s
  • comm utility operations:
    • Intersection: The intersection operation will print the lines that the specified files have in common with one another.
    • Difference: The difference operation will print the lines that the specified files contain and that are not the same in all of those files
    • Set difference: The set difference operation will print the lines in file “A” that do not match those in all of the set of files specified (“B” plus “C” for example)
  • Comparaison:
    $ cat A.txt
    apple
    orange
    gold
    silver
    steel
    iron
    
    $ cat B.txt
    orange
    gold
    cookies
    carrot
    
    $ sort A.txt -o A.txt ; sort B.txt -o B.txt
    $ comm A.txt B.txt 
    apple
            carrot
            cookies
                    gold
    iron
                    orange
    silver
    steel
  • To print intersection we remove columns 1 and 2:
    $ comm A.txt B.txt -1 -2
    gold
    orange
  • To print the lines that are uncommon in the two files:
    $ comm A.txt B.txt  -3
    apple
            carrot
            cookies
    iron
    silver
    steel
    
    # To produce a single column:
    $ comm A.txt B.txt  -3 | sed 's/^\t//'
    apple
    carrot
    cookies
    iron
    silver
    steel
    
    # Difference for A.txt:
    $ comm A.txt B.txt -2 -3
    
    # Difference for B.txt:
    $ comm A.txt B.txt -1 -3
  • Script to remove duplicate files by content:
    #!/bin/bash
    #Filename: remove_duplicates.sh
    #Description:  Find and remove duplicate files and keep one sample 
    of each file.
    ls -lS --time-style=long-iso | awk 'BEGIN { 
      getline; getline; 
      name1=$8; size=$5 
    } 
    {
      name2=$8; 
      if (size==$5) 
      { 
        "md5sum "name1 | getline; csum1=$1;
        "md5sum "name2 | getline; csum2=$1;
        if ( csum1==csum2 ) 
        {
          print name1; print name2
        }
      };
      size=$5; name1=name2; 
    }' | sort -u > duplicate_files 
    cat duplicate_files | xargs -I {} md5sum {} | sort | uniq -w 32 | 
    awk '{ print "^"$2"$" }' | sort -u >  duplicate_sample
    echo Removing...
    comm duplicate_files duplicate_sample  -2 -3 | tee /dev/stderr | 
    xargs rm
    echo Removed duplicates files successfully.
  • We have the setuid bit for the owner: -rwS------
  • We have the setgid bit for the group: ----rwS---
  • We have the sticky bit for directories: -------rwT
  • Set sticky bit:
    $ chmod a+t directory_name
  • Make a file immutable with:
    # chattr +i file
  • Remove immutable state with:
    # chattr -i file
  • Create blank file with:
    $ touch filename
  • Generate files in bulk:
    for name in {1..100}.txt
    do
      touch $name
    done
    
    # We can use different patterns such as:
    test{1..200}.c
    testing{a..z}.txt
  • We can modify only access time or modification time with touch:
    touch -a filename
    touch -m filename
  • We can specify the time to use:
    $ touch -d "Fri Jun 25 20:50:14 IST 1999" filename
  • Create symlink with:
    $ ln -s target symbolic_link_name
  • Print only symlinks:
    $ ls -l | grep "^l"
  • Find symlinks:
    $ find . -type l -print
  • Read the target of a symlink:
    $ readlink web
    /var/www
  • To print the type of a file:
    $ file filename
    
    # For instance:
    $ file /etc/passwd
    /etc/passwd: ASCII text
    
    # To exclude the filename:
    $ file -b /etc/passwd
    ASCII text
  • Script for file statistics:
    #!/bin/bash
    # Filename: filestat.sh
    if [ $# -ne 1 ];
    then
      echo "Usage is $0 basepath";
      exit
    fi
    path=$1
    declare -A statarray;
    while read line;
    do
      ftype=`file -b "$line" | cut -d, -f1`
      let statarray["$ftype"]++;
    done < <(find $path -type f -print)
    echo ============ File types and counts =============
    for ftype in "${!statarray[@]}";
    do
      echo $ftype :  ${statarray["$ftype"]}
    done
  • Usage of the previous script:
    $ ./filetype.sh /home/slynux/programs
    ============ File types and counts =============
    Vim swap file : 1
    ELF 32-bit LSB executable : 6
    ASCII text : 2
    ASCII C program text : 10
  • in Bash 3.x we could have used:
    done <<< "`find $path -type f -print`"
  • Create a 1 GB file:
    $ dd if=/dev/zero of=loobackfile.img bs=1G count=1
    1024+0 records in
    1024+0 records out
    1073741824 bytes (1.1 GB) copied, 37.3155 s, 28.8 MB/s
  • Then format the file:
    $ mkfs.ext4 loopbackfile.img
    
    # Check the file type:
    $ file loobackfile.img
    loobackfile.img: Linux rev 1.0 ext4 filesystem data, 
    UUID=c9d56c42-f8e6-4cbd-aeab-369d5056660a (extents) (large files) 
    (huge files)
  • Mount the file:
    # mkdir /mnt/loopback
    # mount -o loop loopbackfile.img /mnt/loopback
    
    # Or manually:
    # losetup /dev/loop1 loopbackfile.img
    # mount /dev/loop1 /mnt/loopback
    
    # To unmount:
    # umount mount_point
  • We can create partitions inside loopback images:
    # losetup /dev/loop1 loopback.img
    # fdisk /dev/loop1
  • Quicker way to mount llopback disk images with partitions using kpartx:
    # kpartx -v -a diskimage.img 
    add map loop0p1 (252:0): 0 114688 linear /dev/loop0 8192
    add map loop0p2 (252:1): 0 15628288 linear /dev/loop0 122880
    
    # Then
    # mount /dev/mapper/loop0p1 /mnt/disk1
    
    # Then remove the mapping:
    # kpartx -d diskimage.img 
    loop deleted : /dev/loop0
  • Mounting ISO files as loopback:
    # mkdir /mnt/iso
    # mount -o loop linux.iso /mnt/iso
  • Flush changes immediately:
    $ sync
  • To create ISO from cdrom we use:
    # cat /dev/cdrom > image.iso
  • But preferred method is:
    # dd if=/dev/cdrom of=image.iso
  • We can create an iso from a folder with:
    $ mkisofs -V "Label" -o image.iso source_dir/
  • Convert standard ISO to hybrid ISO:
    # isohybrid image.iso
    
    # Then to write the iso on an USB device:
    # dd if=image.iso of=/dev/sdb1
    
    # or:
    # cat image.iso >> /dev/sdb1
    
      * Burning ISO from command line:<code># cdrecord -v dev=/dev/cdrom image.iso
    
    # extra options:
    -speed SPEED # speed in number of x (eg. 8x or 16x for instance)
    -multi # for multi session.
    
    # eject cd rom:
    $ eject
    $ eject -t # to close the tray
  • Say we have the file contents:
    # File 1: version1.txt
      this is the original text
      line2
      line3
      line4
      happy hacking !
    
    # File 2: version2.txt
      this is the original text 
      line2
      line4
      happy hacking ! 
      GNU is not UNIX
      
  • Nonunified diff will be:
    $ diff version1.txt version2.txt 
    3d2
    <line3
    6c5
    > GNU is not UNIX
  • Unified diff output will be:
    $ diff -u version1.txt version2.txt
    --- version1.txt  2010-06-27 10:26:54.384884455 +0530 
    +++ version2.txt  2010-06-27 10:27:28.782140889 +0530 
    @@ -1,5 +1,5 @@ 
    this is the original text 
    line2
    -line3
    line4
    happy hacking !
    -
    +GNU is not UNIX
  • Generate a patch file by redirecting the output of diff to a file:
    $ diff -u version1.txt version2.txt > version.patch
  • Apply a patch with patch command:
    $ patch -p1 version1.txt < version.patch
    patching file version1.txt
  • To revert the patch:
    $ patch -p1 version1.txt < version.patch 
    patching file version1.txt
    Reversed (or previously applied) patch detected!  Assume -R? [n] y
    #Changes are reverted.
    
    # We could use the -R flag directly here to avoid the message.
  • Generate difference between directories:
    $ diff -Naur directory1 directory2
    
    # -N: treating absend files as empty
    # -a : consider all files as text files
    # -u produce unified output
    # -r recursively traverse directories
  • Usage of head:
    # print the first 10 lines:
    $ head file
    
    # read data from stdin:
    $ cat text | head
    
    # specify the number of lines to display:
    $ head -b 4 file
    
    # print all lines except last M lines:
    $ head -n -M file
  • Usage of tail:
    # print the last 10 lines:
    $ tail file
    
    # read from stdin:
    $ cat text | tail
    
    # print the last 5 lines:
    $ tail -n 5 file
    
    # print all lines except first M lines:
    $ tail -n -(M+1)
    
    # Monitor the change in the file content with:
    $ tail -f growing_file
    
    # terminate tail when a process dies:
    $ PID=$(pidof Foo)
    $ tail -f file --pid $PID
  • techniques to list directories:
    $ ls -d */
    
    $ ls -F | grep "/$"
    
    $ ls -l | grep "^d"
    
    $ find . -type d -maxdepth 1 -print
  • We use pushd and popd as:
    pushd /var/www
    # then the stack contains [/var/www | ~ ]
    # and current path is /var/www
    
    # push additional paths:
    pushd /usr/src
    
    # list all the paths in the stack:
    $ dirs
    /usr/src /var/www ~ /usr/share /etc
    
    # switching to a given path in the list (indexes start from 0):
    $ pushd +3
    
    # To remove the last pushed and change to the previous directory:
    $ popd
    
    # remove a specific path from the stack: 
    popd +num # indexes from 0.
  • When we deal only with 2 paths, then we can use instead:
    $ cd /var/www
    $ cd /usr/src
    $ cd - # switch to alternate path.
  • Usage of wc utility:
    # Count number of lines:
    $ wc -l file
    
    # use stdin:
    $ cat file | wc -l
    
    # count num words:
    $ wc -w file
    $ cat file | wc -w
    
    # count num characters:
    $ wc -c file
    $ cat file | wc -c
    
    # print num lines words and characters:
    $ wc file
       1435   15763  112200
    
    # print the length of longest line:
    $ wc file -L
    205
  • with the tree command:
    $ tree some_folder
    
    # We can also provide a pattern:
    $ tree path -P PATTERN # Pattern should be wildcard
    $ tree path -I PATTERN # only files excluding the match pattern
    $ tree -h # print the size along the files and directories.
  • HTML output with tree:
    tree PATH -H http://localhost -o out.html
  • public/books/linux_shell_scripting_cookbook/chapter_3.txt
  • Last modified: 2020/07/10 12:11
  • by 127.0.0.1