HDD parameters

To list all block devices, run:

lsblk

To list all partitions, run:

 fdisk -l

Linux Hard Disk Format Command

Partition the new disk using fdisk command

Fist list all partitions, run:

 fdisk -l

To partition the disk – /dev/sda, enter:

 fdisk /dev/sda  

Then select to create a new partition:

 n

Select partition type:

 p   

Set the desired partition number (be careful not to overlap with previous partitions) and the size.

At the end write the new partition table:

 w   

Format the new disk using mkfs.ext4 command

To format Linux partitions using ext4fs on the new disk:

 mkfs.ext4 /dev/sda1   

Mount the new disk using mount command

First create a mount point /mnt/extradisk1 and use mount command to mount /dev/sda1, enter:

 mkdir /mnt/extradisk1
 mount /dev/sda1 /mnt/extradisk1
 df -H      

Update /etc/fstab file

Open /etc/fstab file, enter:

 vi /etc/fstab

Append as follows:

 /dev/sda1               /mnt/extradisk1           ext4    defaults        1 2

File/Folders

Copy Files/Folders

Copy (and synchronize) with rsync
rsync -azP FILE_SRC FILE_DEST
rsync -azP FOLDER_SOURCE FOLDER_DEST  # will create FOLDER_SOURCE inside FOLDER_DEST (if it does not exist), and will copy the content of FOLDER_SOURCE inside it
rsync -azP FOLDER_SOURCE/ FOLDER_DEST  # will copy the content of FOLDER_SOURCE inside FOLDER_DEST
# For nii.gz files, no need to further compress so -z can be dropped
Copy from remote station via scp
scp username@hostname:/path_to_file

File manipulations

Delete non-empty folder
rm -rf
Delete a bunch of files
find . -name "dti*" -delete

or the more complicated version:

find . -name "dti*" | while read F; do rm $F; done

When you are trying to delete too many files using rm, you may get error message: /bin/rm Argument list too long. Use xargs to avoid this problem.

find ~ -name ‘*.log’ -print0 | xargs -0 rm -f
Rename files automatically
ls *.JPG | while read F; do mv $F beijing_2012-06__$F; done

Do it recursively:

find . -name "t2_seg.nii.gz" -exec bash -c 'mv $(dirname $1)/$(basename $1) $(dirname $1)/t2_seg_manual.nii.gz' -- {} \;
Do something on files modified for the past 10 days
find . -type f -name '*.*' -mtime +10 -exec echo "do something on this file: {}" \;
Set created/modification date on a file
touch -mt YYYYMMDDhhmm file

On Maverick and later, the creation date is not updated if newer than the existing. So you should use:

SetFile -d '11/05/2015 17:00:00' my_file

Size of a disk/folder

du -sh folder_name

or for all folders in the path

du -sh *
du -sm * | sort -nr  # in MB and reverse-ordered by size
Get space on a disk
df -h .

Number of files

== Get number of files that match a pattern

ls -dq *pattern* | wc -l
Get number of files in a folder (recursively)
find .//. ! -name . -print | grep -c //

only counts files modified for the past 24h:

find .//. ! -name . -mtime -1 -print | grep -c //
List files modified for the past 24h
find . -mtime -1 -print

List number of files per folder

find . -maxdepth 1 -mindepth 1 -type d -exec sh -c 'echo "{} : $(find "{}" -type f | wc -l)" file\(s\)' \;

Permissions

Change permission
chmod 644           make a file readable by anyone and writable by the owner only.
chmod 755		  make a file readable/executable by everyone and writable by the owner only.
chmod 701		  r/w/x for the owner, no access for everyone
Change owner of a file
sudo chown erik:staff Desktop/test1.docx
Look for group owner & permission
ls -le@a
Find most recently changed files (less than 1 day ago)
find  -mtime -1 -ls
Search files

Files with specific string inside:

find . -name "string"

Files that have been modified for the past 24 hours:

find ~/Documents -type f -ctime -0 | more
Delete folder
rm -rf "folder"

Stdout / Stderr

Compression/Extraction

Tar

compress:

tar -czf /path/to/output/folder/filename.tar.gz /path/to/folder

extract:

tar -zxvf filename.tar.gz
zip

compress folder:

zip -r archive.zip folder/

# Exclude a sub-folder:
zip -r archive.zip folder/ -x '*subfoldertoexclude*'

extract:

unzip archive.zip
copie de fichiers ds une directory

find -name sica*.png | xargs -t -i /bin/cp ./{} ./images

Checksum

This procedure creates a unique signature for your files and folders. It enables to check for integrity when you share data.

On OSX:

find -s FOLDER -type f -exec shasum {} \; | shasum
find -s FOLDER -type f -exec md5 {} \; | md5

On Linux:

find FOLDER -type f -exec md5sum {} \; | md5sum

Remove files from tmp

find . -name "tmp.*" -type d -print0 | xargs -0 /bin/rm -rf

.bash_profile

launched when you open a new terminal. Configure your environment variables from there. Location: your home folder.

To launch it, type:

source ~/.bash_profile

Basic UNIX commands

Check running processes
top
send email
echo "something" | mailx -s "subject" someone@email.com

Processes

Display processes
ps aux
kill a process based on PID
kill -9 "PID"
kill a process from a user
pkill -U USER  

Internet / Network

Download file from internet

curl -o filename -L URL
# Example for OSF file (note the "?action=download" added after the URL):
curl -o data.zip -L https://osf.io/76jkx/?action=download

Alternatively:

wget -O data URL

Copy file between computers

(thank you Nick for the links!)

Using gist.github.com (only for files <100MB):

1. make a new gist
2. note its ID in its URL (something like 3daa207ea45c75722bd0e3bc914dce3a)
3. `git clone git@github.com:3daa207ea45c75722bd0e3bc914dce3a`
4. `cd 3daa207ea45c75722bd0e3bc914dce3a`
5. add your large file;
6. `git add .; git commit; git push`

Basic commands

List all stations on the network (only works on a server)
findsmb
find DNS
cat /etc/resolv.conf
lookup DNS
host HOST_NAME
host IP_ADDRESS
Clear DNS cache (on OSX 10.8 and later)
sudo killall -HUP mDNSResponder
Connect to another station
ssh IP
or:
ssh username@station.domain
Copy files from a remote station
scp username@station.domain:/path_to_file/name_file .

to copy folder: add “-r”

Screen (for background processes)

Let's say you connect to a station from your laptop and you wish to launch a script that will run for several hours. If you close your laptop, the remote script will stop. To prevent this, use screen. It opens a virtual environment from a remote station, so that any script launched within this environment will continue running even if you close your laptop.

Step-by-step procedure:

  1. connect to a station via ssh
  2. launch screen. It will create a new screen attached to the station.
  3. Do whatever you want (e.g., launch a long process).

Detach you from the screen

screen -d
or type: CTRL+a d

Attach to a detached screen

screen -r

Attach to a not detached screen. (Multi display mode).

screen -x

List of your screens

screen -ls

Kill a screen

screen -X kill  (if you only have one screen running)
screen -X -S [session # you want to kill] kill

Give specific name to a screen session

screen -S name_of_session

VIM

Simple but great editor. Usually installed everywhere.

:w pour sauvegarder :q pour quitter

Colored syntax:

vi ~/.vimrc
add: syntax on

BASH SCRIPT

To put at the beginning of your script

#!/bin/bash
pass argument from file
mycommand "$(< file.txt)"

Arrays

Create array from list of files

FILES=($(ls -r data*.*))

Use array (just an example)

${FILES[2]}

Dates

To list the date every week for the next five weeks, starting in 2 days:

for i in {1..5}; do echo $(date -v +${i}w -v +2d "+%Y-%m-%d"); done

FOR loop

Loop across list elements
ARRAY=(element1 element2 element3)
for i in ${ARRAY[@]}; do
  echo $i
done
Loop across list elements (using indices)
ARRAY=(element1 element2 element3)
for i in ${!ARRAY[@]}; do
  echo "Element: $((i+1))/${#ARRAY[@]}"
  echo ${ARRAY[$i]}
done
Loop across a series of files
for filename in im1 im2 im3 ; do
  bet $filename ${filename}_brain ;
done
Using ls
FILES=`ls folder/*.nii`
for file in $FILES; do
  echo $file
done

Do it in a single line:

FILES=`ls folder/*.nii`; for file in $FILES; do echo $file; done
Using find (recursive)
FILES=`find . -name *.nii`; for file in $FILES; do echo $file; done
Using a list of string
SUBJECT_LIST="errsm_20 errsm_21"

for subject in $SUBJECT_LIST; do
  echo $subject
done

IF statements

AND:

if [ ! -z "$var" ] && [ -e "$var" ]; then
  do something ...
else
  do something else...
fi

alternative:

[ ! -z "$var" -a  -e "$var" ]

OR:

if [ ! -z "$var" ] || [ -e "$var" ]; then
  do something ...
fi

alternative:

[ ! -z "$var" -o  -e "$var" ]

Parse CVS file

while IFS=, read col1 col2 col3 col4 col5
  do
    echo "$col2"
  done < dice.csv

Passing input arguments

#!/bin/bash
# use predefined variables to access passed arguments
#echo arguments to the shell
echo $1 $2 $3 ' -> echo $1 $2 $3'
# list all arguments:
echo $@
# We can also store arguments from bash command line in special array
args=("$@")
#echo arguments to the shell
echo ${args[0]} ${args[1]} ${args[2]} ' -> args=("$@"); echo ${args[0]} ${args[1]} ${args[2]}'
#use $@ to print out all arguments at once
echo $@ ' -> echo $@'
# use $# variable to print out number of arguments passed to the bash script
echo Number of arguments passed: $# ' -> echo Number of arguments passed: $#'  

also see: Bash: parsing arguments with 'getopts'

Test existence of a file

if [ -e "$var" ]; then

Generate log file

LOGFILE=log_my_script.log
{
    #bash commands...
    
} 2>&1 | tee $LOGFILE

Generate a log file of a script directely in terminal:

script.$$ [options] | tee log_file_name.log

Add date to log file: log_file_name_$(date +%Y%m%d).log

color echo

the Mountain Lion Terminal does not support the command echo -e therefore the following has to be used:

printf "\e[0;32mText here\e[0m"

List of colors:

Black        0;30
Blue         0;34
Green        0;32
Cyan         0;36
Red          0;31
Purple       0;35
Brown/Orange 0;33     
Yellow       1;33
Light Gray   0;37
White        1;37
NC='\e[0m'

For bold, replace 0 by 1.

Strings manipulation and batch actions (grep, sed, ...)

Get the first 2 characters of a string
pouf="hola"
echo ${pouf:0:2}
Get the last 2 characters of a string
echo ${pouf: -2}
Remove the first character of a string
echo ${pouf:1}
Remove the last 2 characters of a string
echo ${pouf%??}
Display the first line where a pattern is found (here dim3), and remove the first characters that matches a given string
fslhd t1w.nii.gz | grep -m 1 dim3 | sed -e "s/^dim3           //"
Batch actions with grep

`grep` is powerful for creating an action based on a series of input (e.g. multiple files). Below is an example to delete several branches in git, based on their prefix name:

git branch -D `git branch | grep -E 'BRANCH_PREFIX*'`
Check if substring is in string
string='My long string'
if [[ $string == *"My long"* ]]; then
  echo "It's there!"
fi

More info here: http://tldp.org/LDP/abs/html/string-manipulation.html