CLI

Table of Contents

• Binutils & Coreutils
  • File utilities
• Text utilities
• Shell utilities
  • Binary utilities
  • Shell
  • File management
  • Editing
  • Coding
  • Desktop
  • Task management
  • Music
  • Internet
• Shell Commands
  • Interesting Aliases
  • Redirecting output to other terminal
  • Invoke an editor to write a long, complex, or tricky command
  • List of commands you use most often
  • Check command history, but avoid running it
  • Run the last command as root
  • Create a script of the last executed command
• Files
  • Like top, but for files
  • Empty a file
  • Zeroing out the empty space and making deleted information non-recoverable
  • Copy folder structure without files
  • Find all extensions in a directory tree
  • Tail a file with less
• Text
  • Trim trailing whitespace and expand tabs to spaces in a tree
  • Convert between DOS / UNIX textfile formats
  • Diff two unsorted files without creating temporary files
  • ASCII table
• System
  • Display which distro is installed
  • Currently mounted filesystems
  • Execute a command at a given time
  • Mount a temporary ram partition
• Memory
  • See all strings in RAM
  • Display the top ten running processes sorted by memory usage
• Networking
  • Set audible alarm when an ip address comes online.
  • Serve files locally
  • Port forwarding
  • Network monitoring
  • SSH configuration
  • Pipe over network

Binutils & Coreutils

File utilities

Text utilities

Shell utilities

Binary utilities

Shell

• zsh
• tmux
• htop
• glances

File management

• mc
• ranger
• nnn
• tmsu
• fdupes
• ncdu
• rsync

Editing

• vim
• neovim
• emacs
• kak
• sed
• awk

Coding

• jrnl
• cloc
• git
• tig
• jsonpp
• jq
• grep
• ack
• ag
• ripgrep
• gdb
• rr
• valgrind

Desktop

• dwm
• i3
• bspwm
• awesome

Task management

• taskwarrior
• timewarrior
• vit
• tasksh

Music

• mpd
• ncmpcpp

Internet

• mosh
• weechat
• rtv

Shell Commands

* In bash, 'ctrl-r' searches your command history as you type
- Add "set -o vi" in your ~/.bashrc to make use the vi keybindings instead
  of the Emacs ones. Takes some time to get used to, but it's fantastic!
- Input from the commandline as if it were a file by replacing
  'command < file.in' with 'command <<< "some input text"'
- '^' is a sed-like operator to replace chars from last command
  'ls docs; ^docs^web^' is equal to 'ls web'. The second argument can be empty.
* '!!:n' selects the nth argument of the last command, and '!$' the last arg
  'ls file1 file2 file3; cat !!:1-2' shows all files and cats only 1 and 2
- More in-line substitutions: http://tiny.cc/ecv0cw http://tiny.cc/8zbltw
- 'nohup ./long_script &' to leave stuff in background even if you logout
- 'cd -' change to the previous directory you were working on
- 'ctrl-x ctrl-e' opens an editor to work with long or complex command lines
* Use traps for cleaning up bash scripts on exit http://tiny.cc/traps
* 'shopt -s cdspell' automatically fixes your 'cd folder' spelling mistakes
- 'dict' is a commandline dictionary
* 'trash-cli' sends files to the trash instead of deleting them forever.
  Be very careful with 'rm' or maybe make a wrapper to avoid deleting '*' by
  accident (e.g. you want to type 'rm tmp*' but type 'rm tmp *')
- 'file' gives information about a file, as image dimensions or text encoding
- 'echo start_backup.sh | at midnight' starts a command at the specified time
- Pipe any command over 'column -t' to nicely align the columns
* Google 'magic sysrq' and learn how to bring you machine back from the dead
- 'diff --side-by-side fileA.txt fileB.txt | pager' to see a nice diff
* 'j.py' http://tiny.cc/62qjow remembers your most used folders and is an
  incredible substitute to browse directories by name instead of 'cd'
- 'dropbox_uploader.sh' http://tiny.cc/o2qjow is a fantastic solution to
  upload by commandline via Dropbox's API if you can't use the official client
- learn to use 'pushd' to save time navigating folders (j.py is better though)
- if you liked the 'psgrep' alias, check 'pgrep' as it is far more powerful
* never run 'chmod o+x * -R', capitalize the X to avoid executable files. If
  you want _only_ executable folders: 'find . -type d -exec chmod g+x {} \;'
- 'xargs' gets its input from a pipe and runs some command for each argument

Interesting Aliases

• function lt() { ls -ltrsa "$@" | tail; }
• function psgrep() { ps axuf | grep -v grep | grep "$@" -i --color=auto; }
• function fname() { find . -iname "*$@*"; }

Redirecting output to other terminal

There are many reasons one might want to see output from shell commands in another terminal emulator but it definitely has its uses. The other day it just so happended that I needed such a functionality. Without going into details I’m going to show you how to achieve such behaviour – and more – easily, by leveraging the fact that under the hood std(in|out|err) are just unix file descriptors.

All that’s needed is the snippet below

exec 1>/proc/<PID>/fd/1

Where <PID> is id of the process you want to redirect your output to (and you can obtain it for instance by running echo $$ inside the target terminal).

But don’t take my word for it – let’s see for ourselves how it works.

exec me

I think the biggest revelation comes with acknowledging the fact that exec is used for something more than to (according to man): "replace the shell with command without creating a new process"

Because this functionality isn’t even the first thing mentioned in man. The first thing man has to say about exec is that

The exec utility shall open, close, and/or copy file descriptors as specified by
any redirections as part of the command.

And before you ask – yes, there’s a caveat. Unfortunately (or fortunately for process security), once the process has started there’s no way to change its file descriptors, unless you resort to some gdb wizardry – and even then it’s not always possible.

Wrapping up – while it’s not really a game-changer, it’s still a feature that can sometimes be put to good use.

Invoke an editor to write a long, complex, or tricky command

$ ctrl-x e

Next time you are using your shell, try typing ctrl-x e (that is holding control key press x and then e). The shell will take what you've written on the command line thus far and paste it into the editor specified by $EDITOR. Then you can edit at leisure using all the powerful macros and commands of vi, emacs, nano, or whatever.

List of commands you use most often

$ history | awk $({a[$2]++}END{for(i in a){print a[i] " " i}}) | sort -rn | head

Check command history, but avoid running it

$ !whatever:p

!whatever will search your command history and execute the first command that matches 'whatever' If you don’t feel safe doing this put :p on the end to print without executing. Recommended when running as superuser.

Run the last command as root

$ sudo !!

Useful when you forget to use sudo for a command. !! grabs the last run command. Similarly !$ gets the last argument of the previous command And !<#> repeats that line number from bash history, so:

$ history
332   ls -l
$ !332
ls -l

Along the same line $_. Repeat the last argument.

ping mybox.domain
ssh $_

You can also append it so something like:

ping mybox.domain
scp file $_:

Meta period (esc-.) cycles the last argument from the previous commands. It works on the current prompt.

Create a script of the last executed command

$ echo "!!" > foo.sh

Sometimes commands are long, but useful, so it's helpful to be able to make them permanent without having to retype them.

Files

Like top, but for files

$ watch -d -n 2 $(df; ls -FlAt;)

Empty a file

$ > file.txt

For when you want to flush all content from a file without removing it.

Zeroing out the empty space and making deleted information non-recoverable

$ sudo su
$ touch junk.bin # Create a dummy file named junk.bin on the USB flash drive
$ fdisk -l and locate the partition you would like to zero out
$ mkdir /tmp/ddusb
$ mount -o loop /dev/sdxx /tmp/ddsdb (replacing sdxxx with your partition)
$ dd if=/dev/zero of=/tmp/ddusb/junk.bin
$ rm /tmp/ddusb/junk.bin

Copy folder structure without files

$ cd /new/dir
$ (cd /old/dir; find -type d ! -name . -printf "\"%p\"\n") | xargs mkdir

"\"%p\"\n" Double quoted path names or else xargs and mkdir will fail.

Find all extensions in a directory tree

$ find . -type f | awk -F . '{print $NF}' | tr '[:upper:]' '[:lower:]' | sort | uniq

Tail a file with less

$ less +F somelogfile

Using +F will put less in follow mode. This works similar to 'tail -f' To stop scrolling, use the interrupt. Then you’ll get the normal benefits of less (scroll, etc.). Pressing SHIFT-F will resume the 'tailing'.

Text

Trim trailing whitespace and expand tabs to spaces in a tree

$ find . -type f \( -name "*.ex" -o -name "*.exs" \) -exec bash -c 'echo "$0" && sed -i "" -e "s/[[:space:]]*$//" "$0" && expand -t 4 "$0" > /tmp/e && mv /tmp/e "$0"' {} \;

The command finds all erlang and elixir source files and expands tabs to 4 spaces and removes trailing whitespace.

• -type f \( -name "*.erl" -o -name "*.ex" -o -name "*.exs" \) filters all erlang and elixir source files. The escaped parentheses are necessary to tell find that the -name filters are part of one condition, otherwise it will short-circuit evaluate them.
• -exec bash -c tell find to run a bash command on each matching file.
• echo "$0" will print out the first parameter (i.e. the filename) passed to the bash command.
• sed -i "" -e "s/[[:space:]]*$//" "$0" will run sed in-place on the file, telling it to match all whitespace at the end of line and replace it with nothing.
• expand -t 4 "$0" > /tmp/e && mv /tmp/e "$0" will run expand on the file, since expand does not have an option to run it in-place we redirect output to a temporary file and then copy it over the original file.

Convert between DOS / UNIX textfile formats

$ cp <fileid> <fileid>.dos &&
$ cat <fileid>.dos | tr -d '\r' > <fileid>

Diff two unsorted files without creating temporary files

$ diff <(sort file1) <(sort file2)

bash subshell redirection (as file descriptors) used as input to diff

ASCII table

$ man ascii

System

Display which distro is installed

cat /etc/issue

Currently mounted filesystems

$ mount | column -t

Particularly useful if you're mounting different drives, using the following command will allow you to see all the filesystems currently mounted on your computer and their respective specs with the added benefit of nice formatting.

Execute a command at a given time

$ echo "ls -l" | at midnight

This is an alternative to cron which allows a one-off task to be scheduled for a certain time.

Mount a temporary ram partition

$ mount -t tmpfs tmpfs /mnt -o size=1024m

Makes a partition in ram which is useful if you need a temporary working space as read/write access is fast. Be aware that anything saved in this partition will be gone after your computer is turned off.

Memory

See all strings in RAM

$ sudo dd if=/dev/mem | cat | strings

This command will show you all the string (plain text) values in ram. A fun thing to do with ram is actually open it up and take a peek.

Display the top ten running processes sorted by memory usage

ps aux | sort -nk +4 | tail

ps returns all running processes which are then sorted by the 4th field in numerical order and the top 10 are sent to STDOUT.

Networking

Set audible alarm when an ip address comes online.

$ ping -i 60 -a IP_address

Waiting for your server to finish rebooting? Issue the command above and you will hear a beep when it comes online. The -i 60 flag tells ping to wait for 60 seconds between ping, putting less strain on your system. Vary it to your need. The -a flag tells ping to include an audible bell in the output when a package is received (that is, when your server comes online).

Serve files locally

python -m SimpleHTTPServer 8080 shares all the files in the current folder over HTTP, port 8080

Port forwarding

ssh -R 12345:localhost:22 server.com "sleep 1000; exit"

forwards server.com's port 12345 to your local ssh port, even if you machine is not externally visible on the net. Now you can ssh localhost -p 12345 from server.com and you will log into your machine. 'sleep' avoids getting kicked out from server.com for inactivity

Network monitoring

Some tools to monitor network connections and bandwith:

• lsof -i monitors network connections in real time
• iftop shows bandwith usage per connection
• nethogs shows the bandwith usage per process

SSH configuration

Use this trick on .ssh/config to directly access 'host2' which is on a private network, and must be accessed by ssh-ing into 'host1' first

  Host host2
      ProxyCommand ssh -T host1 'nc %h %p'
  	  HostName host2

Pipe over network

Pipe a compressed file over ssh to avoid creating large temporary .tgz files

tar cz folder/ | ssh server "tar xz"