alias Returns all active aliases
To add default aliases, write in your .bashrc or .profile files alias=='' Example: alias l='ls -la'
unalias Remove from alias list
Execute a bash script at a given time Example: at -f <commands.sh> 21.00
Trims paths and extensions froms strings (opposite of DIRNAME)
Example: FASTA returns panorthologs/famcl00140.faa basename $FASTA returns famcl00140.faa basename $FASTA .faa returns famcl00140
see also DIRNAME
Calculator echo "100/50" | bc Yields 2 echo "scale=2:100/60 | bc -l Yields 1.67
-e Show newline characters as '$'
-A Show tabs (^I), newline characters (
-A Shows identity of whitespace characters. Much nicer than cat -A --config-file Show location of your bat config file
Neat trick: cat > new_file Then type whatever texts you want to type End with Ctrl-D You now have a new file with the typed content!
column -t [file] Prints the file so that all columns are nicely aligned in view column -t [file] -s $'\t' Use TAB as separator instead of the default whitespace You can also type 'Ctrl-v TAB' while typing the comment. "
cd - Returns to the previous directory
Change rights of owner / user, group, others or all of a file to either read, write or execute. Example: chmod u+x Adds the e(x)ecutable right for (u)ser to this chmod ug+x Add executable rights to user and group chmod u-x Remove executable rights for user chmod 755 7=4+2+1=r+w+x for user. 7=4+2+1=r+w+x for group. 5=4+1=r+x for others chmod 600 6=4+2=r+w for user, 0=--- for group, 0=--- for others
Change ownership of a given directory Example: chown kasiazar kasia/ -R Recursive (any directory below will also be changed ownership)
Compares to files and lists the common lines and unique files Files have to be sorted first (in the same way ofcourse) They have to be sorted lexicographically apparently (-V sort on both files won't work with comm) Column1: unique to file1 '-1' to suppress this column Column2: unique to file2 '-2' to suppress this column Column3: appear in both files '-3' to suppress this column
cp -P [source] [target] Copy a symbolic link
Converts between image file formats Example: convert filename.pdf filename.png convert -resize 200x100 filename.pdf filename.png Resizes the image while converting convert -resize 200 filename.pdf filename.png Resizes the image to 200px wide, but keeps aspect ratio for length convert -append *.png output.png Merge images into a single file, stack images vertically convert +append *.png output.png Merge images into a single file, stack images horizontally
Convert xslx to csv Example: in2csv in.xlsx > out.csv Install with 'brew install csvkit'
-l List all current crontabs -e Edit crontab commands crontab command format: [minute] [hour] [day of the month] [week] [month] [day of the week] Example 0 12 * * * * Every day, at 12.00
# general syntax
cut [flags] [file]
# select the 3rd column, using space as a delimiter
cut -f 3 -d' '
# select columns 1, 3 or 4
cut -f 1,3,4 -d' '
# select columns 1-5
cut -f 1-5 -d' '
# select columns 1-3 and 5-7
cut -f 1-3,5-7 -d ' '
# select columns 3 to the last column
cut -f 3- -d ' '
# set a literal single tick as delimiter
cut -d\'
# print all columns except the 3rd column
cut -f 3 --complement
# instead of columns, you can only simply print the n'th character in each line
# print the 17th character in each line
cut -c 17
# trick to print only the last column
# NOTE: you may need to put cut -f 2 instead
cmd | rev | cut -f 1 | revdatamash -s -g 1 collapse 2 A 1 A 2 B 3 C 4 D 5 D 6
into
A 1,2 B 3 C 4 D 5,6
Shows differences between two files diff
-y invoke side-by-side mode, really nice -w set width of visibles lines of differing lines in side by side mode --suppress-common-lines --color=always color persists to less -R
Example: diff -W 200 -y --color=always | less -R
Trims filenames from strings (opposite of BASENAME)
Example: FILE returns /path/to/panorthologs/famcl00140.faa dirname $FILE returns /path/to/panorthologs
DNSDOMAINNAME dnsdomainname Reveals the domain name of the current computer where you are logged in Example: icm.uu.se
-h <file|directory> Human readable numbers du -h . | sort -h -r | head -n 20 Nice quick command to look for the top biggest directories from current directory du -sh
Report total size of dir only. Will not report any nested directories or filesecho "$STRING" Prints string to STDOUT echo $STRING Same thing but converts tabs to spaces
Options: -e Enables interpretation of backslash escapes and certain characters are recognized, like \t and \n -n Removes the newline character
Takes a string as an argument and evaluates it as if you'd typed that string on a command line
exit Exit with an exit code. Any non-zero exit code will be interpreted as exited with an error.
find <PATH> <flags> \;
-name <EXPRESSION>
-maxdepth <number> The level of directories down find will try to find your attern
-size 0
-exec <cmd> {} {} refers to each file that was found
-xtype l Find broken symlinks
-newermt "2022-05-10" Find files that are newer (modified later) than May 10th 2022
-ls Print found files like an 'ls -l' print
Example:
find . -name "All.htm"
find . -size 0 -exec rm {} \; Remove empty files
find . -xtype l -exec rm {} \; Remove broken symlinks
# sort all files in directory and subdirectories by modification time
find . -type f -printf '%TY-%Tm-%Td %TH:%TM: %Tz %p\n' | sort -n
# list all directories that do not contain a file that has pattern
find . -type d ! -exec sh -c 'ls "{}" | grep -q "PATTERN"' \; -printFG Bring back jobs that are running in the background, into the "foreground". Jobs that are running in the background can be displayed with "jobs" Ctrl-Z to put current windown in the background. Then you can use 'fg' to put it back to the foreground
GREP grep "PATTERN" OUTPUTS THE ENTIRE LINE OF WHERE IT FINDS ITS HITS grep -v "^$" REMOVES EMPTY LINES (no special character, but clever use of regexp: ^ start of line, $ end of line) Flags: -i Ignores case -n REPORTS LINE NUMBER OF HITS -c COUNTS HITS -c "" Counts lines in the file -A PRINTS IN ADDITION TO HIT LINE ALSO int LINES AFTER HIT -v PRINTS ALL LINES EXCEPT LINES WITH HITS -P Allows for Perl regexp in PATTERN -f Prints all lines that have the PATTERNS listed in FILE, one per line -x The PATTERN must encompass the entire line. Handy for speeding up grep --no-group-separator Suppresses the '--' seperator when using -A, -B, or -C -o If you have multiple hits in a single line, it will report all the hits on that line, seperated by newlines. It will not report the whole line, but just the match. Hence the name '-only_matching' --color=always Keep coloring of match even if you pipe into a new grep command. Even less keeps color Can even do 2 different colors. Example echo "hello" | grep --color=always "he" | GREP_COLORS="mt=01;34" grep "lo" -e Multiple search patterns at the same time. grep -e "pattern1" -e "pattern2" -m Only reports the first lines with match -H Always reports the filename that it found its hit in -q Quiet mode
..AGREP Grep that allows mismatching. Very handy to search sequence data still allowing sequence errors --color -E Allowed of mismatches
A modern faster replacement of grep written in Rust. Particularly much faster than grep when using a pattern file (-f)
If no target file is given, it will search all files recursively down from the current directory
# prints all lines that have the PATTERNS listed in FILE, one per line. MUCCCH FASTER than grep -f
rg -f [pattern_file]
# NOTE: if pattern_file contains patterns with dots, pipes, carots, backslashes etc, they will be interpreted as regular expressions
# if you want to interpret them literally, use -F
# if you want only some of them to be interpreted literally, don't use -F and escape only those characters that you want to interpret literally in the pattern file
# suppress line numbers
rg -N
# count hits
rg -c
# search compressed files
rg -z
# ensures that color highlighting persists when for example piping to less -SR
rg --color=always
# hides filenames when reporting lines that match expression
rg --no-filename
# literal search
# treat dots and special characters that could be interpreted as regular expressions literally
rg -F, --fixed-strings
# only report lines where the search pattern fits an entire word in the target files line
# so for example if query is '123', it will hit ' 123 ' and not ' 1234 '. It will however still match '123.'
rg -w, --word-regexp
# find all files that do NOT contain a certain expression
rg --files-without-match 'Best FunDi parameter' *log
# only report that part of the line that matches the regexp
rg -o "p.*_exca.*$"
rg --only-matching "p.*_exca.*$"HOSTNAME hostname Reveals the name of the host Example: molev209
IDENTIFY Reports properties of an image file. Part of the ImageMagick suite. Example: identify filename.png
Reports:
filename.png PNG 200x3510 200x3510+0+0 16-bit DirectClass 1.267MB 0.000u 0:00.000
JOBS Lists all 'jobs', that can either be commands, or scripts or anything like that in the background. Putting an '&' sign after the job command puts it in the background. With FG (foreground), you put the 'job' back in the foreground. Then you can kill it for example. JOBS only lists those jobs that are running in that specific terminal
Join two files based on a common field between two files to create a new, merged file NOTE: FILE1 and FILE2 need to be sorted (on their common field) prior to attempting to join!
# complicated example
join -a 1 -a 2 -e '0' -1 2 -2 2 -o '1.1,2.1,0' -t $'\t' FILE1 FILE2 > FILE3
# Switch to print unpairable fields of FILE1 and FILE2
-a 1 -a 2
# Replace missing fields with 0 (if a column value exists in FILE1 and not in FILE2 or vice versa)
-e '0'
# Join using the 2nd field in file 1 and 2nd field in file 2
-1 2 -2 2
# Print the field 1 of file 1, field 1 of file 2, and the field used to join them (0 special meaning)
-o '1.1,2.1,0'
# Set field delimiter. Tab is default, but by setting this explicitly, Tab becomes the delimiter in the output too
-t $'\t'
KILL Killing defunct processes. For example when Firefox crashes ps -e | grep firefox kill -9 <process_id_of_firefox>
LESS /PATTERN Search for regex pattern in file from this point on forward ?PATTERN Search for regex pattern in file from this point on backwards Press 'n' to go to the next hit Press 'N' to go the previous hit then g Go to line
Flags: -S Show file without line breaks -N Show line numbers -R Show font colors +F
LN
Symbolic links
ln -s
LS ls -S Sorts all files in directory by size ls -t Sorts all files in directory by last time modified ls -d */ List only directories ls -L List the file that the symbolic link is pointing to ls --ignore= List all files except those that match
ls --full-time
LSOF
lsof | grep '.nfs00000000107c4e6200248a94'
MAN man -l Will open the file with man man ./ Will open the file with man
# Create nested directories in a single command. 'p' stands for parent directory
# -p also prevents the 'does already exists' error if directory already exists
mkdir -p newdir1/newdir2
# Create directory with these permissions ??
mkdir -m 777tmp=$(mktemp) mktemp returns a random temporary file name and creates that file tmp_dir=$(mktemp -d) mktemp -d returns a random temporary directory name and creates that directory
more Opens the file for reading. Can for some files display colors, underlines and bold text
This is a nice way to get the permissions of all directories leading up to your file and the file itself
$ namei -l /scratch3/jmartijn/ergo-genome/results/46_EditedPolishedGenome/Ergobibamus_cyprinoides_CL.scaffolds.edited.fasta
f: /scratch3/jmartijn/ergo-genome/results/46_EditedPolishedGenome/Ergobibamus_cyprinoides_CL.scaffolds.edited.fasta
drwxr-xr-x root root /
drwxr-xr-x root root scratch3
drwxr-x--x jmartijn roger jmartijn
drwxr-xr-x jmartijn roger ergo-genome
drwxr-xr-x jmartijn roger results
drwxr-xr-x jmartijn roger 46_EditedPolishedGenome
-rw-r--r-- jmartijn roger Ergobibamus_cyprinoides_CL.scaffolds.edited.fastanohub Continue running the program even if the session ends
Basic usage
# feed in a glob referencing all the files you want to process
parallel -j[n_processors] "[command you want to parallelize] {}" ::: [files to process]
# also works with single quotes
parallel -j[n_processors] '[command you want to parallelize] {}' ::: [files to process]
# feed in a file that references all the files you want to process
parallel -j[n_processors] "[command you want to parallelize] {}" :::: [file_containing_patterns_to_process]
# storing variables inside a parallel command is possible
# it seems important to use single quotes for outer, and inner quotes for inner
parallel -j10 'LL=$(rg "FunDi log-likelihood" {}.log | cut -f3 -d" "); echo {} "$LL"' :::: ../leca_fams.list
# process all commands listed in $JOBS with parallel
parallel -j2 < $(JOBS)
# prevent new jobs from being invoked if one of the processed jobs returns an error
parallel -j2 --halt soon,fail=1 < $(JOBS)Example
parallel -j8 "seqret -sequence <(zcat {}) -outseq fasta/{/.}.fasta" ::: ../../data/wgs_*.dat.gzThis command will use 8 processors, so 8 seqret commands will be executed at the same time until all files have been processed.
{} holds the name of the file, including the path, for example in this case ../../data/wgs_cevg01_env.dat.gz {/.} holds the basname of the file, and the extension trimmed, for example in this case wgs_cevg01_env.dat If you don't provide a number of processors, parallel will take ALL available processors
Currently running jobs are still finished properly
paste Merge lines of files
# join all the lines in a file into a single line, using tab as delimiter
paste -s /path/to/file
# join all the lines into a single line, using a custom delimiter
paste -s -d ',' /path/to/file
# merge two files side by side
paste /path/to/file1 /path/to/file2
# example usecase
cat [list] | paste -sd+ | bc easy code to get the sum of column of numbersAs a replacement for sed. Perl -PE has a better regular expressions support -p assumes the code 'while () { ##code## } ' -e anything within '' will be seen as a perl script
Example: perl -pe 's:^.+$:TEST: if /^>/' aaa_pooled-hfix.fasta -i Changes input file. (Just like sed). But the -i has to be first flag for some reason -w use warnings
Concatenate PDFs pdfunite *.pdf out.pdf or pdftk *.pdf cat output <new.pdf>
Subset pages of a PDF pdftk <in.pdf> cat 1-4 6-7 output <out.pdf> Put pages 1-4 and 6-7 of <in.pdf> into <out.pdf> pdftk <in.pdf> cat 8-end output <out.pdf> Put pages 8 until the last page into <out.pdf>
Concatenate PNGs
# concatenate left to right
magick convert +append one.png two.png three.png out.png
# concatenate top to bottom
magick convert -append one.png two.png three.png out.pngps Shows a list of active processes with process IDs (PIDs) and the command used to start these processes ps -fu $USER Shows all actives processes invoked by $USER ps -o <field1,field2,etc> Show process key fields. Example: ps -o user,uid,comm,pid,pcpu,tty ps -r Sort by pcpu (%CPU)
PPID: Parent Process ID
readlink -f <file.txt> Returns the absolute path of the file If realpath is not available
realpath <file.txt> Returns the absolute path of the file
reboots the system
rsync [option] [source] [destination]
# -- common options
rsync -avzh --progress
# -a Archive mode, keeps all symlinks, devices, attributes, permissions, ownerships etc.
# -v Verbose mode
# -z Compress data during transfer
# -h Human readable output
# --progress Shows progress bars of transfer
# Tells rsync to copy directories recursively
rsync -r
rsync --recursive
# make your rsync command 'restartable',
# such that if it breaks or crashes for whatever reason you can restart it later
rsync --partial
# combine --partial and --progress
rsync -P
# only copy files from the source that do not already exist in the target
rsync --ignore-existing
# -- include or exclude certain files
# Rsync all files except for those that end with .txt
rsync --exclude='*.txt'
# Exclude all files but do NOT exclude those ending with .txt
rsync --exclude='*' --include='*.txt'
# Rsync only the files that are listed in 'files.txt'.
rsync --files-from='files.txt'
# Example: rsync --files-from=files.list jmartijn@molev-32-72.icm.uu.se:/path/of/destination/ .
# Where files.list contained only file names, no paths or slashes etc.
# Rsync all files except those listed in files.txt
rsync --exclude-from='files.txt'
# NOTE: Make sure the filenames in files.txt are relative to <source>, not necessarily from where you execute the command!
# -- miscellaneous rsync options
# Creates log of rsync transfer
rsync --log-file=[logfile]
# Do a dry-run. Execute a test operation without making any changes
rsync --dry-run
# Deletes files in the destination that are no long present in the source
rsync --delete-after
# Rsyncs the file that the link is pointed to
rsync -L
rsync --copy-links
# Skip files that are newer on the receiver
rsync -u or --update
# Preserve permissions
rsync -p
# Update destination files in-place. Update file without creating a new file.
rsync --inplace
# Without --inplace, rsync would create a new tmp file, copy the updated parts onto it, swap with destination file, delete old copy of destination file
# Append data onto shorter files
rsync --append
# Append assumes that the destination file is a shorter version of the source file, meaning the destination file is identical to the start of the source file
# -- example use-cases
# For updating phylobayes chain backups, this command is really useful
rsync -avP --inplace --append chain*.* chain_backup/screen Open up a new screen screen -S Create a new screen with screen -ls List down the current open screens screen -r Reattach to the screen screen -r Reattach to screen with the set sessionname screen -r If there is only one screen, it automatically will reattach to that one. exit Exit the current screen you are in (this will kill any processses in this screen?) killall screen Kill all screens
WHILE IN SCREEN ctrl-a + ctrl-d Detach from the screen while not interuppting any processes in the screen ctrl-a + ':' + sessionname + Enter Rename a session to something easy to remember ctrl-a + Esc You can now scroll up and down using arrow keys and PgUp PgDwn
SEQ Creates a sequence of numbers seq -s seq -w 000 050 500 returns 000 050 100 150 etc. -w ensures that all numbers have the same digit length.
Example: END=5 seq 0 $END Returns 0 1 2 3 4 5, separated by
SIPS
Rotate png / jpg images a certain number of degrees
sips
-r <degrees_rotating_clockwise>
--padColor <color_of_resulting_deadspace> (FFFFFF = white)
--out <out_file>
If you want to rotate counter clockwise e.g. 10 degrees, simply rotate clockwise 360-10=350 degrees.
# Set field separator to underscore
sort -t "_"
# Reverses the sort output (for example instead of large to small go from small to large)
sort -r
# Omits lines that are copies in the first field after sorting
sort -u -k 1,1
# Sort numerical lists that include exponential values (e.g. ending with e-06)
sort -g
# Sort human readable numbers, incompatible with -n
# when they include K, M, G, T etc
# e.g. 100M, 512M
sort -h
# All fields from field 1 until the last field
sort -k 1
# Just field 1
sort -k 1,1
# Sort field 1 numerically
sort -k 1,1n
# Sort the file in place
sort -o FILE FILE
# Sort based on the numbers in a field, also if it contains letters and dots etc. Version sort
sort -VSHUTDOWN shutdown -r +5 "Rebooting soon, log out ASAP!" Gives people who are logged in some time to log out.
SPLIT Split a file by size, by lines or other things split -l 10000 Splits up a file into multiple files with 10000 lines each split -d Makes numerical suffixes instead of alphabetical split -n l/ Splits file into chunks while not splitting lines
SSH Login as root: Type either 'su' and then enter the root password or ssh into your own machine as root as follows: ssh -i .ssh/root_rsa root@molev209 and enter the passphrase for key '.ssh/root_rsa' (not the same as the root password) ssh -X enable X11-forwarding ssh -Y enable trusted X11-forwarding Once you are in the server, you can check if X11 server is active with echo $DISPLAY which should return something like 'localhost:15.0'
add ssh key so you only need to type passphrase once per session ssh-add ~/.ssh/perun_rsa
SVN cd desired/directory svn checkout svn://foo DOWNLOADS REPOSITORY, THIS IS ONLY NEEDED TO DO ONCE Before you can add a new file to the svn, you actually need to physically copy it to the svn folder. svn add [new dir] svn add [new file] svn commit -m "Message to the subversion system DATE" svn update UDPATES YOUR LOCAL REPOSITORY WITH LATEST CHANGES ON SUBVERSION SYSTEM
TAIL / HEAD tail Shows the last 10 lines of a file - Shows the last [number] lines of a file -n Shows the last [number] lines of a file -n + Shows the last part of a file starting at line [number] -f Keeps updating, if the file is being appended continuously
head Shows the first 10 lines of a file - Shows the first lines of a file -n Shows the first lines of a file -n - Shows the first part of a file until line
note that tail -n+ and head -n- only work with GNU head/tail ?
TAR tar is short for "tape archive" tar -tvf Lists the contents of the .tar tar -ztvf Lists the contents of the .tar.gz tar -jtvf Lists the contents of the .tar.bz2 tar -czvf .tar.gz /path/to/dir-or-file> (Create) an archive with g(Z)ip, with (V)erbose and specify (F)ilename tar xzf <tar_ball> -C <output_dir>
TR
# If you have multiple white space characters in a row, it will truncate it to just a single white space
tr -s ' '
# Transform all tab into newlines
tr '\t' '\n'
# Transform all a's into d's, b's into e's etc.
tr '[abc]' '[def]'
# Deletes space ánd >
tr -d ' >'
# Removes all newline characters
# effectively converts multiple lines into a single line string
tr -d '\n'
# Counts number of '(' in file
cat FILE | tr -dc '(' | wc -cTIME time [cmd] Will print out the time it takes to execute a given command
UNAME Prints system information -a Prints all information
Prints among other stuff, hostname, 32-bit or 64-bit
UNIQ -u show which lines are not repeated (?) -d show only duplicate lines
VI(M) :x Quit
W Shows you who is currently logged in
WGET wget <source_url> -P <out_dir>
WC wc -m Prints the number of characters in the entire file
WHATIS whatis Gives a oneline summary of what the is
WHEREIS whereis Searches standard unix/linux locations, like /usr/local/bin, /usr/bin, /bin/ etc for Shows ALL locations of where a tool is installed. For instance whereis python can show you all different python installations
WHICH
which Searches your $PATH for the executable that is and returns the directori(es) in your $PATH that have said
which -a Lists all locations at which command is installed
perl -pe 'if (/^>/) {my @line = split /|/,$_; s/.+/>$line[3]/;}' 16S_Pooled_id83-otu-0.89-pruned.fasta
perl -pe 's/)([1-9]|[1-6][0-9]):/):/g' newick > newick-edit.out
perl -pe "s/)\d+:/):/g" newick > newick-edit.out
cut -f 5 | tr '\t' '+' | sed 's/+$/\n/' | bc
sed -n 'n;p;n;n' <.fastq> | tr -d "\n" | wc -c
sort -k1,1 -k12,12gr -k11,11g -k3,3gr contigs_vs_nt.blastn | sort -u -k1,1 --merge 1,1 query name 12,12 bit score 11,11 evalue 3,3 id
sort -k1,1 -k15,15nr -k12,12nr batch-PDB-results.exhaustive.pooled.txt | sort -u -k1,1 --merge > batch-PDB-results.exhaustive.pooled.besthits.txt
sed -i -r "s/)([0-9]{1,2}).[0-9]:/)\1:/g"