Stothard Research Group

http://www.ualberta.ca/~stothard/software.html

align_learn.pl - this Perl script converts a multiple sequence alignment into a format that can be readily analyzed using common machine learning algorithms. Specifically, the program accepts a sequence alignment in FASTA format and converts it into an ARFF (attribute-relation file format) file containing data attributes and data instances. The ARFF format can then be read by the Weka machine learning software, which provides implementations of many machine learning algorithms.
Developed by: Paul Stothard.
Availability: align_learn.zip, align_learn.tar.gz.

annotate_SNPs.pl - this Perl script annotates SNPs identified by the next-generation sequencing of genomic DNA or transcripts. It is designed to accept SNPs from the AB diBayes SNP package, Maq, or any other program that can provide a reference sequence identifier, a SNP position, the base found at that position in the reference sequence, and the bases found at that position among the sequencing reads. The script examines each SNP record provided as input and uses Ensembl, NCBI, and EBI to provide a detailed description of its expected functional significance, regardless of whether or not the SNP has been previously described. Included in this description is a functional class ('NON_SYNONYMOUS_CODING' for example), and information about the affected transcript, if applicable, including transcript ID, position of the SNP, the alleles as they would appear in the transcript sequence, and Gene Ontology information. For protein-altering SNPs, each of the resulting protein sequences is compared to orthologous proteins, to determine which of the alleles most drastically changes the protein. Using a locally installed version of Ensembl this script can annotate 4,000,000 SNPs in about two days on a standard desktop system. Thus this script is suitable for annotating the SNPs arising from genome resequencing projects.
Developed by: Paul Stothard.
Availability: Included in the NGS-SNP package.

backup.sh - this shell script archives directories of interest on a Linux-based system. When it is first run, and on the first of each month, this script generates a full backup of the files and directories listed in the include.conf file. Files and directories listed in the exclude.conf file are not included in the archive. These full backups are not overwritten by future backups. Each Sunday the script performs a full backup that is overwritten the following Sunday. Every day the script performs an incremental backup, storing the files that have changed since the last full backup. These incremental backup files are named after the day of the week they are performed, and are overwritten each week. The script sends an email on the first of each month, and whenever any backup fails. The script splits each full backup into a series of smaller files, suitable for burning to CD or DVD. When a full backup is generated, the MD5 hash value of the complete backup file is written to a README file in the same directory as the backup files. Included in the README are directions for assembling the split backup files into the original file.
Developed by: Paul Stothard.
Availability: backup_script.zip, backup_script.tar.gz.

blast_hit_features.pl - this Perl script accepts BLAST results obtained from local_blast_client.pl or remote_blast_client.pl. The results must have been obtained using blastn, tblastn, or tblastx searches (i.e. nucleotide databases), since this script uses GenBank files to obtain feature information for sequence hits. For each entry in the BLAST results, the GI number of the hit, if available, is used to obtain the corresponding sequence record from NCBI, in GenBank format. The features in the GenBank file are compared to the coordinates of the HSP, and features overlapping with the HSP are added to the existing BLAST results. The modified results are written to a new file. The three nearest features preceding the HSP (located to the left of the HSP) and the three nearest features located after the HSP (located to the right of the HSP) are also added to the output.
Developed by: Paul Stothard.
Availability: blast_hit_features.zip, blast_hit_features.tar.gz.

blast_hit_flanking_sequence.pl - this Perl script accepts blastn search results obtained from local_blast_client.pl or remote_blast_client.pl. In addition to the BLAST results file, the script requires the query sequences and database sequences in FASTA format. For each BLAST result, the script constructs a modified query sequence, in which the query is extended using sequence extracted from the hit sequence. The amount of hit sequence added to the ends of the query can be specified using the -u and -d options.
Developed by: Paul Stothard.
Availability: blast_hit_flanking_sequence.zip, blast_hit_flanking_sequence.tar.gz.

blast_hits_in_ucsc_genome_browser.pl - this Perl script accepts BLAST results obtained from local_blast_client.pl or remote_blast_client.pl. The results must have been obtained using blastn, tblastn, or tblastx searches (i.e. nucleotide databases) and database sequences downloaded from the UCSC Genome Browser site (http://hgdownload.cse.ucsc.edu/downloads.html). The BLAST results are converted to annotation files for the UCSC Genome Browser, and a separate HTML file containing links to each feature in the annotation files is created. Clicking on a link in the HTML file loads the genome region involving the BLAST HSP into the UCSC Genome Browser and passes the annotations in the relevant annotation file to the browser for inclusion in the view.
Developed by: Paul Stothard.
Availability: blast_hits_in_ucsc_genome_browser.zip, blast_hits_in_ucsc_genome_browser.tar.gz.

build_cluster_script.pl - this Perl script creates an executable shell script with the specified command repeated n number of times. Every occurrence of the '$' symbol in the command is replaced by a number, from 1 to n. Alternatively, the "-l" option causes letters to be used in place of numbers (eg 'aa' instead of '1', 'ab' instead of '2'). This script can be used to generate scripts for batch processing on a computer cluster.
Developed by: Jason Grant and Paul Stothard.
Availability: build_cluster_script.zip, build_cluster_script.tar.gz.

cDNA_library_entropy.pl - this Perl script accepts a directory containing one or more sequence files in multi-FASTA format. Typically each file will contain the sequences obtained from a single EST library or tissue type. By default the script looks for a UniGene annotation identifier in each sequence title, for example 'Bt.22094'. A different ID type can be specified using the -m option. The number of sequences present for each ID is determined. The script uses these counts to calculate the information entropy of the library in bits. This value increases as the number of distinct sequences in a library increases, and decreases as the number of replicates of a particular sequence increases. The -d option can be used to obtain the information entropy of combinations of libraries. For example, specifying '-d 2' causes all possible combinations of two libraries to be evaluated. This script is intended to aid in the selection of tissues for SNP discovery by mRNA sequencing.
Developed by: Paul Stothard.
Availability: Included in the NGS-SNP package.

CGView - a Java package for generating high quality, zoomable maps of circular genomes. Its primary purpose is to serve as a component of sequence annotation pipelines. Feature information and rendering options are supplied to the program using an XML file, a tab delimited file, or an NCBI ptt file. CGView converts the input into a graphical map (PNG, JPG, or Scalable Vector Graphics format), complete with labels, a title, and legends. In addition to the default full view map, the program can generate a series of hyperlinked maps showing expanded views. The linked maps can be explored using any web browser, allowing rapid genome browsing, and facilitating data sharing. The feature labels in maps can be hyperlinked to external resources, allowing CGView maps to be integrated with existing web site content or databases.
Developed by: Paul Stothard.
Availability: http://bioinformatics.org/cgview/

CGView Comparison Tool (CCT) - a package for visually comparing bacterial, plasmid, chloroplast, or mitochondrial sequences of interest to existing genomes or sequence collections. The comparisons are conducted using BLAST, and the BLAST results are presented in the form of graphical maps that can also show sequence features, gene and protein names, COG category assignments, and sequence composition characteristics. CCT can generate maps in a variety of sizes, including 400 Megapixel maps suitable for posters. Comparisons can be conducted within a particular species or genus, or all available genomes can be used. The entire map creation process, from downloading sequences to redrawing zoomed maps, can be completed easily using scripts included with the CCT. User-defined features or analysis results can be included on maps, and maps can be extensively customized. To simplify program setup, a CCT virtual machine that includes all dependencies preinstalled is available. Detailed tutorials illustrating the use of CCT are included with the CCT documentation.
Developed by: Paul Stothard and Jason Grant.
Availability: http://stothard.afns.ualberta.ca/downloads/CCT/

cgview_xml_builder.pl - this Perl script accepts a variety of input files pertaining to circular genomes and generates an XML file for the CGView genome drawing program. This script can create the XML to display a variety of sequence composition plots, gene expression data, COG information, BLAST results, and more. See the included README file for additional information.
Developed by: Paul Stothard.
Availability: cgview_xml_builder.zip, cgview_xml_builder.tar.gz.

combine_output_files.pl - this Perl script combines files that are part of a file series (created by split_records.pl for example). Several options are avialable for controlling how comments and header lines are handled. This script can be used to combine results files generated on a computer cluster.
Developed by: Paul Stothard.
Availability: combine_output_files.zip, combine_output_files.tar.gz.

genome_pattern_search.pl - a Perl program that reads a genomic sequence in FASTA format and searches for the patterns you specify using regular expressions. A summary is generated for each sequence match, including: the sequence fragment that matched the pattern; the position of the first base; the position of the last base; the strand on which the match was found; the name of the gene containing the match or "not in gene"; the name of the nearest downstream gene; a description of the gene; the distance to the nearest downstream gene; the total times this exact sequence was found; the percentage of the instances of this exact sequence that were found inside of genes; and the average number of base pairs to the downstream gene for this exact sequence.
Developed by: Paul Stothard.
Availability: genome_pattern_search.zip, genome_pattern_search.tar.gz.

get_cds.pl - this Perl script accepts a GenBank or EMBL file and extracts the protein translations or the DNA coding sequences and writes them to a new file in FASTA format. Information indicating the reading frame and position of the coding sequence relative to the source sequence is added to the titles.
Developed by: Paul Stothard.
Availability: get_cds.zip, get_cds.tar.gz.

get_genes_in_area.pl - this Perl script accepts as input a position or list of positions in a genome and returns descriptions of nearby genes. The descriptions include position and function information, along with identifiers that can be used to access related records in other databases.
Developed by: Paul Stothard.
Availability: Included in the NGS-SNP package.

get_orfs.pl - this Perl script accepts a sequence file as input and extracts the open reading frames (ORFs) greater than or equal to the size you specify. The resulting ORFs can be returned as DNA sequences or as protein sequences. The titles of the sequences include start, stop, strand, and reading frame information. The sequence numbering includes the stop codon (when encountered) but the translations do not include a stop codon character.
Developed by: Paul Stothard.
Availability: get_orfs.zip, get_orfs.tar.gz.

get_snps_by_gene_ontology.pl - this Perl script accepts a species name and a Gene Ontology (GO) accession number, and returns a list of SNPs located in or nearby genes associated with the GO accession. Several fields of information are provided for each SNP, including ID, location, flanking sequence, and alleles. Gene and transcript identifiers and descriptions of gene function are also provided.
Developed by: Paul Stothard.
Availability: get_snps_by_gene_ontology.zip, get_snps_by_gene_ontology.tar.gz.

local_blast_client.pl - this Perl script accepts a FASTA file containing multiple sequences as input. It then submits each sequence to a locally installed version of the blastall program. For each of the hits obtained, the script retrieves a descriptive title by performing a separate Entrez search of NCBI's databases. Each BLAST hit and its descriptive title are written to a single tab-delimited output file.
Developed by: Paul Stothard.
Availability: local_blast_client.zip, local_blast_client.tar.gz.

maq_pipeline.sh - this bash script processes short sequence reads from Illumina's Genome Analyzer (Solexa) system, using the Maq package. The script automates the entire analysis process, and parallelizes the most intensive analysis step if run on a computing cluster with Sun Grid Engine.
Developed by: Paul Stothard.
Availability: maq_pipeline.zip, maq_pipeline.tar.gz.

md5_sums.pl - this Perl script accepts a list of directories and recursively generates a list of the files in the directories and their MD5 values. An optional list of directories and files to exclude from the calculation can also be supplied. The MD5 calculation can be skipped for large files, using the optional size parameter.
Developed by: Jason Grant.
Availability: md5_sums.zip, md5_sums.tar.gz.

ncbi_monitor.pl - this Perl script performs NCBI Entrez searches to identify publications related to genomic regions of interest in a species of interest. More specifically, this script accepts an organism name, chromosome name, and base position as input. It then retrieves the IDs for all Entrez Gene records located within a certain distance of the base position (the distance can be adjusted using the -f option). For each Gene record the script obtains IDs of PubMed records identified by NCBI as being related to the Gene record. If the script has previously written output to the specified output directory (i.e. the directory supplied using the -o option), it examines the previously obtained PubMed IDs to see which IDs are new. An email message describing the newly obtained records is then sent to the email address supplied using the -e option. The PubMed results are also written to a file in the output directory. If the -h option is specified, NCBI's HomoloGene database is also queried for each Gene record, in an attempt to obtain additional PubMed records, linked to the HomoloGene hits. These PubMed records may describe results obtained in other species, but could be relevant nonetheless.
Developed by: Paul Stothard.
Availability: Included in the NGS-SNP package.

ncbi_search.pl - this Perl script uses NCBI's Entrez Programming Utilities to perform searches of NCBI databases. The script can return complete database records, or sequence IDs.
Developed by: Paul Stothard.
Availability: ncbi_search.zip, ncbi_search.tar.gz.

NGS-SNP - this collection of scripts annotates raw SNP lists returned from programs such as Maq. SNPs are classified as synonymous, nonsynonymous, 3' UTR, etc. regardless of whether or not they match existing SNP records. Included among the annotations, several of which are not available from any existing SNP annotation tools, are the results of detailed comparisons with orthologous sequences. These comparisons allow, for example, SNPs to be sorted or filtered based on how drastically the SNP changes the score of a protein alignment. Other fields indicate whether or not the SNP-altered residue exhibits co-evolution with other residues in the protein, the names of overlapping protein domains or features, and the conservation of both the SNP site and flanking regions. NCBI, Ensembl, and Uniprot IDs are provided for genes, transcripts, and proteins when applicable, along with Gene Ontology terms, a gene description, phenotypes linked to the gene, and an indication of whether the SNP is novel or known. A "Model_Annotations" field provides several annotations obtained by transferring in silico the SNP to an orthologous gene, typically in a well-characterized species.
Developed by: Paul Stothard, Jason Grant, and Xiaoping Liao.
Availability: NGS-SNP.

obtain_reference_transcripts.pl - this Perl script builds a FASTA file consisting of the canonical transcripts for all the genes in Ensembl for a given organism. The canonical transcript is defined as either the longest CDS (if the gene encodes a protein), or as the longest cDNA. Ensembl gene entries can be associated with many transcripts--this script aims to get the "best" single transcript for each gene. The resulting FASTA file is suitable for sequence searches and for mapping sequence reads derived from cDNAs. The -a option can be used to specify that all transcripts should be downloaded, not just the canonical ones.
Developed by: Paul Stothard.
Availability: Included in the NGS-SNP package.

obtain_reference_chromosomes.pl - this Perl script builds a FASTA file consisting of the chromosome sequences in Ensembl for a given organism. The resulting FASTA file is suitable for sequence searches and for mapping sequence reads.
Developed by: Paul Stothard.
Availability: Included in the NGS-SNP package.

random_sequence_reads.pl - this Perl script generates simulated sequence reads from a file of sequences in FASTA format. The starting position of each read is chosen at random. The length of the reads is specified using the -L option. Reads truncated because the end of a sequence is encountered are discarded if they are shorter than the length specified using the -m option. Sampling is continued until the desired number of reads is obtained.
Developed by: Paul Stothard.
Availability: random_sequence_reads.zip, random_sequence_reads.tar.gz.

random_sequence_sample.pl - this Perl script selects a random sample of sequences from a FASTA file containing multiple sequences. The sample is written to a new text file. Sampling can be performed with or without replacement.
Developed by: Paul Stothard.
Availability: random_sequence_sample.zip, random_sequence_sample.tar.gz.

remote_blast_client.pl - this Perl script accepts a FASTA file containing multiple sequences as input. It submits each sequence to NCBI's BLAST servers, to identify related sequences in a database of interest. An optional 'limit by entrez query' value can be supplied to restrict the search. For each BLAST hit a descriptive title is obtained using a separate Entrez search. Each BLAST hit and its descriptive title are written to a single tab-delimited output file.
Developed by: Paul Stothard.
Availability: remote_blast_client.zip, remote_blast_client.tar.gz.

remote_in_silico_pcr.rb - this Ruby script accepts as input a list of primer sequences and uses the remote "UCSC In-Silico PCR" site to perform in silico PCR on the specified genome. By default only the top hit is returned for each primer pair--all the hits can be returned by using the '-m' option.
Developed by: Jason Grant.
Availability: remote_in_silico_pcr.zip, remote_in_silico_pcr.tar.gz.

sequence_to_multi_fasta.pl - this Perl script reads a file consisting of a single DNA sequence (in raw, FASTA, GenBank, or EMBL format) and then divides the sequence into smaller sequences of the size you specify. The new sequences are written to a single output file with a modified title giving the position of the subsequence in relation to the original sequence. The new sequences are written in FASTA format.
Developed by: Paul Stothard.
Availability: sequence_to_multi_fasta.zip, sequence_to_multi_fasta.tar.gz.

space_check.sh - this shell script monitors hard drive space and sends an email when space becomes scarce. On the first day of each month the script sends an email report of hard drive space.
Developed by: Paul Stothard.
Availability: space_check.zip, space_check.tar.gz.

split_records.pl - this Perl script splits an input file into multiple output files, to allow analysis jobs to be divided among nodes in a computer cluster Several options are included for handling header lines, for specifying the record separator, and for controlling how files are named.
Developed by: Jason Grant and Paul Stothard.
Availability: split_records.zip, split_records.tar.gz.

Sebas Ramos-Onsins lab

http://bioinformatics.cragenomica.es/numgenomics/people/sebas/software/software.html

Analysis of Nucleotide Variability on Natural and Domesticated Populations:

Our focus is mainly on the species Sus scrofa (the pig) and close related species. We are interested in explaining the evolutionary processes involved in the history of this species and the effect of natural and artificial selection in wild and domestic breeds.

We are also interested in analyzing the variability in autopolyploid species using statistics adapted for pooled lineages.

Study and development of Neutrality tests and methods for statistical inference of evolutionary models:

A useful way to obtain a more interpretable information from raw sequencing data is the use of statistics and neutrality tests. They summarize the information observed and make possible to understand the evolutionary patterns more easily. We study and develop neutrality test to be used in evolutionary parameter inference.

Development of tools for the analysis of nucleotide variability:

Until now we have been working in developing tools for the analysis of nucleotide variability in multilocus data: analysis on multilocus data can be much precise in comparison to single locus because they reduce the variance of the parameters inferred in the analysis. We develop tools for multilocus population genetic analysis.

Now we are focusing on population genomics: massive parallel sequencing is revolutionizing the study of population genetics in many ways. We are developing bioinformatic tools for the analysis of variability at genomics level.

Genome-Wide Survey of Pseudogenes in 80 Fully Re-sequenced Arabidopsis thaliana Accessions

Pseudogenes (Ψs), including processed and non-processed Ψs, are ubiquitous genetic elements derived from originally functional genes in all studied genomes within the three kingdoms of life. However, systematic surveys of non-processed Ψs utilizing genomic information from multiple samples within a species are still rare. Here a systematic comparative analysis was conducted of Ψs within 80 fully re-sequenced Arabidopsis thalianaaccessions, and 7546 genes, representing ~28% of the genomic annotated open reading frames (ORFs), were found with disruptive mutations in at least one accession. The distribution of these Ψs on chromosomes showed a significantly negative correlation betweenΨs/ORFs and their local gene densities, suggesting a higher proportion of Ψs in gene desert regions, e.g. near centromeres. On the other hand, compared with the non-Ψ loci, even the intact coding sequences (CDSs) in the Ψ loci were found to have shorter CDS length, fewer exon number and lower GC content. In addition, a significant functional bias against the null hypothesis was detected in the Ψs mainly involved in responses to environmental stimuli and biotic stress as reported, suggesting that they are likely important for adaptive evolution to rapidly changing environments by pseudogenization to accumulate successive mutations.

Microindel detection in short-read sequence data - comparing BWA with other aligners

The sensitivity of indel detection based on Novoalign and especially BWA alignments benefits from higher sequencing depth and read length.

http://bioinformatics.oxfordjournals.org/content/26/6/722.full

Eckertlab - on evolutionary genomics of trees

http://eckertlab.blogspot.ca/

the Eckert Lab located in the Department of Biology at Virginia Commonwealth University. 
A plant evolutionary genomics lab. 

Research topics range from the dissection of adaptive plant phenotypes into their genetic components to inferences of genome-wide patterns of polymorphism, divergence and natural selection.

http://evolfri.blogspot.ca/

good GUI tool for sequences analysis

1. Unipro UGENE

http://ugene.unipro.ru/

http://bioinformatics.oxfordjournals.org/content/28/8/1166

• CAP3
• Bowtie-0.12.7
• BWA-0.5.9
• blast-2.2.25
• blast-2.2.25+
• clustalw-2.1
• mafft-6.847
• T-Coffee-8.99
• Mr.Bayes-3.2.0

Summary: Unipro UGENE is a multiplatform open-source software with the main goal of assisting molecular biologists without much expertise in bioinformatics to manage, analyze and visualize their data. UGENE integrates widely used bioinformatics tools within a common user interface. The toolkit supports multiple biological data formats and allows the retrieval of data from remote data sources. It provides visualization modules for biological objects such as annotated genome sequences, Next Generation Sequencing (NGS) assembly data, multiple sequence alignments, phylogenetic trees and 3D structures. Most of the integrated algorithms are tuned for maximum performance by the usage of multithreading and special processor instructions. UGENE includes a visual environment for creating reusable workflows that can be launched on local resources or in a High Performance Computing (HPC) environment. UGENE is written in C++ using the Qt framework. The built-in plugin system and structured UGENE API make it possible to extend the toolkit with new functionality.

Availability and implementation: UGENE binaries are freely available for MS Windows, Linux and Mac OS X at http://ugene.unipro.ru/download.html. UGENE code is licensed under the GPLv2; the information about the code licensing and copyright of integrated tools can be found in the LICENSE.3rd_party file provided with the source bundle.

modify sequence name in a fasta file

cat genes.fasta | awk '{if (/^>/) {print $1} else {print $0}}' > genes2.fasta

To shorten the sequence names in fasta file.

change
>sequense1 human person1
ATGC

to

>sequense1
ATGC

Some unix/perl oneliners for Bioinformatics

http://genomics-array.blogspot.ca/2010/11/some-unixperl-oneliners-for.html

1    File format conversion/line counting/counting number of files etc.

1.    $ wc –l   : count number of lines in a file.
2.    $ ls | wc –l        : count number of files in a directory.
3.    $ tac     : print the file in reverse order e.g; last line first, first line last.
4.    $ rev     : reverse the file in lines.
5.    $ sed 's/.$//' or sed 's/^M$//' or sed 's/\x0D$//' : converts a dos file into unix mode.
6.    $sed "s/$/`echo -e \\\r`/" or sed 's/$/\r/' or sed "s/$//": converts a unix newline into a DOS newline.
7.    $ awk '1; { print "" }' : Double space a file.
8.    $ awk '{ total = total + NF }; END { print total+0 }' : prints the number of words in a file.
9.    $sed '/^$/d' or [grep ‘.’] : Delete all blank lines in a file.
10.    $sed '/./,$!d' : Delete all blank lines in the beginning of the file.
11.    $sed -e :a -e '/^\n*$/{$d;N;ba' -e '}': Delete all blank lines at the end of the file.
12.    $sed -e :a -e 's/<[^>]*>//g;/
13.    $sed 's/^[ \t]*//' : deleting all leading white space tabs in a file.
14.    $ sed 's/[ \t]*$//' : Delete all trailing white space and tab in a file.
15.    $ sed 's/^[ \t]*//;s/[ \t]*$//' : Delete both leading and trailing white space and tab in a file.


2.2    Working with Patterns/numbers in a sequence file
16.    $awk '/Pattern/ { n++ }; END { print n+0 }' : print the total number of lines containing the word pattern.
17.    $sed 10q : print first 10 lines.
18.    $sed -n '/regexp/p' : Print the line that matches the pattern.
19.    $sed '/regexp/d' : Deletes the lines that matches the regexp.
20.    $sed -n '/regexp/!p' : Print the lines that does not match the pattern.
21.    $sed '/regexp/!d' : Deletes the lines that does NOT match the regular expression.
22.    $sed -n '/^.\{65\}/p' : print lines that are longer than 65 characters.
23.    $sed -n '/^.\{65\}/!p' : print lines that are lesser than 65 characters.
24.    $sed -n '/regexp/{g;1!p;};h' : print one line before the pattern match.
25.    $sed -n '/regexp/{n;p;}' : print one line after the pattern match.
26.    $sed -n '/^.\{65\}/ {g;1!p;};h' < sojae_seq > tmp : print the names of the sequences that are larger than 65 nucleotide long.
27.    $sed -n '/regexp/,$p' : Print regular expression to the end of file.
28.    $sed -n '8,12p' : print line 8 to 12(inclusive)
29.    $sed -n '52p' : print only line number 52.
30.    $seq ‘/pattern1/,/pattern2/d’ < inputfile > outfile : will delete all the lines between pattern1 and pattern2.
31.    $sed ‘/20,30/d’ < inputfile > outfile : will delete all lines between 20 and 30.   OR sed ‘/20,30/d’ < input > output will delete lines between 20 and 30.
32.    awk '/baz/ { gsub(/foo/, "bar") }; { print }' : Substitute foo with bar in lines that contains ‘baz’.
33.    awk '!/baz/ { gsub(/foo/, "bar") }; { print }' : Substitute foo with bar in lines that does not contain ‘baz’.
34.    grep –i –B 1 ‘pattern’ filename > out : Will print the name of the sequence and the sequence having the pattern in a case insensitive way(make sure the sequence name and the sequence each occupy a single line).
35.    grep –i –A 1 ‘seqname’ filename > out : will print the sequence name as well as the sequence into file ‘out’.  

2.3    Inserting Data into a file:

36.    gawk --re-interval 'BEGIN{ while(a++<49 1="" amp="" filename="" nbsp="" s="" sub="" x="">> fileout : will insert 49 ‘X’ in the sixth position of every line.


37.    gawk --re-interval 'BEGIN{ s="YourName" }; { sub(/^.{6}/,"&" s) }; 1' : Insert your name at the 6 th position in every line.

3.    Working with Data Files[Tab delimited files]:

3.1    Error Checking and data handling:
38.    awk '{ print NF ":" $0 } ' : print the number of fields of each line followed by the line.
39.    awk '{ print $NF }' : print the last field of each line.
40.    awk 'NF > n' : print every line with more than ‘n’ fields.
41.    awk '$NF > n' : print every line where the last field is greater than n.
42.    awk '{ print $2, $1 }' : prints just first 2 fields of a data file in reverse order.
43.    awk '{ temp = $1; $1 = $2; $2 = temp; print }' : prints all the fields in the correct order except the first 2 fields.
44.    awk '{ for (i=NF; i>0; i--) printf("%s ", $i); printf ("\n") }' : prints all the fields in reverse order.
45.    awk '{ $2 = ""; print }' : deletes the 2nd field in each line.
46.    awk '$5 == "abc123"' : print each line where the 5th field is equal to ‘abc123’.
47.    awk '$5 != "abc123"' : print each line where 5th field is NOT equal to abc123.
48.    awk '$7  ~ /^[a-f]/' : Print each line whose 7th field matches the regular expression.
49.    awk '$7 !~ /^[a-f]/' : print each line whose 7th field does NOT match the regular expression.
50.    cut –f n1,n2,n3.. > output file : will cut n1,n2,n3 columns(fields) from input file and print the output in output file. If delimiter is other than TAB then give additional argument such as cut –d ‘,’ –f n1,n2.. inputfile > out
51.    sort –n –k 2,2 –k 4,4 file > fileout : Will conduct a numerical sort of column 2, and then column 4. If –n is not specified, then, sort will do a lexicographical sort(of the ascii value).

4.    Miscellaneous:
52.    uniq –u inputfile > out : will print only the uniq lines present in the sorted input file.
53.    uniq –d inputfile > out : will print only the lines that are in doubles from the sorted input file.
54.    cat file1 file2 file3 … fileN > outfile : Will concatenate files back to back in outfile.
55.    paste file1 file2 > outfile : will merge two files horizontally. This function is good for merging with same number of rows but different column width.
56.    !:p : will print the previous command run with the ‘pattern’ in it.
57.    !! : repeat the last command entered at the shell.
58.    ~ : Go back to home directory
59.     echo {a,t,g,c}{a,t,g,c}{a,t,g,c}{a,t,g,c} : will generate all tetramers using ‘atgc’. If you want pentamers/hexamers etc. then just increase the number of bracketed entities.NOTE: This is not a efficient sequence shuffler. If you wish to generate longer sequences then use other means.
60.    kill -HUP ` ps -aef | grep -i firefox | sort -k 2 -r | sed 1d | awk ' { print $2 } ' ` : Kills a hanging firefox process.
61.    csplit -n 7 input.fasta '/>/' '{*}' : will split the file ‘input.fasta’ wherever it encounters delimiter ‘>’. The file names will appear as 7 digit long strings.
62.    find . -name data.txt –print: finds and prints the path for file data.txt.
Sample Script to make set operations on sequence files:
63.    grep ‘>’ filenameA > list1  # Will list just the sequence names in a file names.
grep ‘>’ filenameB > list2 # Will list names for file 2
cat list1 list2 > tmp # concatenates list1 and list2 into tmp
sort tmp > tmp1 # File sorted
uniq –u tmp1 > uniq    # AUB – A ∩ B (OR (A-B) U (B-A)) 
uniq –d tmp1 > double  # Is the intersection (A ∩ B)
cat uniq double > Union # AUB
cat list1 double > tmp
sort tmp | uniq –u > list1uniq # A - B
cat list2 double > tmp
sort tmp | uniq –u > list2uniq # B - A     



PERL ONELINERS:

1.    perl -pe '$\="\n"'   : double space a file
2.    perl -pe '$_ .= "\n" unless /^$/' : double space a file except blank lines
3.    perl -pe '$_.="\n"x7' : 7 space in a line.
4.    perl -ne 'print unless /^$/' : remove all blank lines
5.    perl -lne 'print if length($_) < 20' : print all lines with length less than 20.
6.    perl -00 -pe '' : If there are multiple spaces, delete all leaving one(make the file a single spaced file).
7.    perl -00 -pe '$_.="\n"x4' : Expand single blank lines into 4 consecutive blank lines
8.    perl -pe '$_ = "$. $_"': Number all lines in a file
9.    perl -pe '$_ = ++$a." $_" if /./' : Number only non-empty lines in a file
10.    perl -ne 'print ++$a." $_" if /./' : Number and print only non-empty lines in a file
11.    perl -pe '$_ = ++$a." $_" if /regex/' ; Number only lines that match a pattern
12.    perl -ne 'print ++$a." $_" if /regex/' : Number and print only lines that match a pattern
13.    perl -ne 'printf "%-5d %s", $., $_ if /regex/' : Left align lines with 5 white spaces if matches a pattern (perl -ne 'printf "%-5d %s", $., $_' : for all the lines)
14.    perl -le 'print scalar(grep{/./}<>)' : prints the total number of non-empty lines in a file
15.    perl -lne '$a++ if /regex/; END {print $a+0}' : print the total number of lines that matches the pattern
16.    perl -alne 'print scalar @F' : print the total number fields(words) in each line.
17.    perl -alne '$t += @F; END { print $t}' : Find total number of words in the file
18.    perl -alne 'map { /regex/ && $t++ } @F; END { print $t }' : find total number of fields that match the pattern
19.    perl -lne '/regex/ && $t++; END { print $t }' : Find total number of lines that match a pattern
20.    perl -le '$n = 20; $m = 35; ($m,$n) = ($n,$m%$n) while $n; print $m' : will calculate the GCD of two numbers.
21.    perl -le '$a = $n = 20; $b = $m = 35; ($m,$n) = ($n,$m%$n) while $n; print $a*$b/$m' : will calculate lcd of 20 and 35.
22.    perl -le '$n=10; $min=5; $max=15; $, = " "; print map { int(rand($max-$min))+$min } 1..$n' : Generates 10 random numbers between 5 and 15.
23.    perl -le 'print map { ("a".."z",”0”..”9”)[rand 36] } 1..8': Generates a 8 character password from a to z and number 0 – 9.
24.    perl -le 'print map { ("a",”t”,”g”,”c”)[rand 4] } 1..20': Generates a 20 nucleotide long random residue.
25.    perl -le 'print "a"x50': generate a string of ‘x’ 50 character long
26.    perl -le 'print join ", ", map { ord } split //, "hello world"': Will print the ascii value of the string hello world.
27.    perl -le '@ascii = (99, 111, 100, 105, 110, 103); print pack("C*", @ascii)': converts ascii values into character strings.
28.    perl -le '@odd = grep {$_ % 2 == 1} 1..100; print "@odd"': Generates an array of odd numbers.
29.    perl -le '@even = grep {$_ % 2 == 0} 1..100; print "@even"': Generate an array of even numbers
30.    perl -lpe 'y/A-Za-z/N-ZA-Mn-za-m/' file: Convert the entire file into 13 characters offset(ROT13)
31.    perl -nle 'print uc' : Convert all text to uppercase:
32.    perl -nle 'print lc' : Convert text to lowercase:
33.    perl -nle 'print ucfirst lc' : Convert only first letter of first word to uppercas
34.    perl -ple 'y/A-Za-z/a-zA-Z/' : Convert upper case to lower case and vice versa
35.    perl -ple 's/(\w+)/\u$1/g' : Camel Casing
36.    perl -pe 's|\n|\r\n|' : Convert unix new lines into DOS new lines:
37.    perl -pe 's|\r\n|\n|' : Convert DOS newlines into unix new line
38.    perl -pe 's|\n|\r|' : Convert unix newlines into MAC newlines:
39.    perl -pe '/regexp/ && s/foo/bar/' : Substitute a foo with a bar in a line with a regexp.

Some other Perl Tricks

Want to display some progress bars while perl does your job:

For this perl provides a nice utility called "pipe opens" ('perldoc -f open' will provide more info)

open(my $file, '-|', 'command','option', 'option', ...) or die "Could not run tar ... - $!";
  while (<$file>) {
       print "-";
  }
  print "\n";
  close($file);

Will print - on the screen till the process is completed 

creating password less connections to multiple remote machines

creating password less connections to multiple remote machines

Many times while working in multiple machines, you may like to automate certain processes where the programs can directly access information/data from another server effortlessly - without a password. This can be done using ssh-keygenprotocol.

So, what happens here is; you have a local machine, lets call it 'A' and you have a remote machine, lets call it as 'B'. You have an account in 'B' and that is say 'myname'. Everytime you log into that machine using ssh, you have to do something like:

$ ssh myname@B

$Password:

$myname@B:

In order to directly log into a machine without a password, you have to generate a pair of keys; called as a public key and a private key. The public key is the public information and the private key is only known to your local machine i.e; 'A'. You can use ssh-keygen to create a pair of keys in a given time. This is how you should proceed:

$ ssh-keygen -t rsa

Generating public/private rsa key pair.
Enter file in which to save the key (/home/Sucheta/.ssh/id_rsa): /home/Sucheta/.
ssh/iicb_rsa [ Remember here to enter a new file name, else it will over write any other id_rsa file that you may have saved earlier for any other computer ]
Enter passphrase (empty for no passphrase): [Enter a paraphrase that is > 4 characters long. This is essential if your computer has more than one pair of public private keys for more than one remote server]
Enter same passphrase again:
Your identification has been saved in /home/Sucheta/.ssh/iicb_rsa.
Your public key has been saved in /home/Sucheta/.ssh/iicb_rsa.pub.
The key fingerprint is:
95:13:96:1b:66:ef:36:74:25:76:05:23:64:58:bb:94 Sucheta@Sucheta-PC
The key's randomart image is:
+--[ RSA 2048]----+
|          o== o.o|
|         .*+ +o.o|
|         o++E. + |
|         ..oo..  |
|        S  o..   |
|            +    |
|           . .   |
|                 |
|                 |
+-----------------+
Then do:

$ssh-copy-id myname@B

[This command will append your public key in the ~/.ssh/authorized_keys file in the remote host. You can also do this manually by logging back to your remote computer and copy pasting your public key in the 'authorized_keys' file. Make sure your public key is copy pasted in one single line.
Another thing to remember is, depending on the OS and version, the file that needs to have the public key in the remote machine may be different. In order to confirm that it is indeed called as "authorized_keys" do the following: 
[root@Apala ssh]# cat /etc/ssh/sshd_config | grep Keys
# HostKeys for protocol version 2
#AuthorizedKeysFile     .ssh/authorized_keys
#AuthorizedKeysCommand none
#AuthorizedKeysCommandRunAs nobody

This tells you indeed the file that stores public key in your remote computer is named as authorized_keys .

Next time you try to create another passwordless connection to another computer, just repeat the above steps. And always remember to write the public and private key into different files, else it will overwrite contents into id_rsa and id_rsa.pub file. Add a passphrase too.

One more important thing to remember is to check the file permission for "authorized_keys" file in the remote machine.Always set it to 700.

Using this, you can also automate file transfer by an sftp or any other remote ftp protocols