How to use awk with multiple files?

file1.txt:

John John2
Apple Apple2
Mike Mike2
Max Max2

file2.txt:

Max
Tim
Tom
Poole
Mike

1. Find identifiers that occur in both files

Code:

awk -F "\t" ' NR==FNR { array[$1]; next} $0 in array {print}' file1.txt file2.txt

The two files are processed sequentially. NR == FNR identifies records from the first non-empty file, allowing its identifiers to populate a lookup table. NR counts records across both files; FNR restarts in each file. For two four-line files, both counters run from 1 to 4 in the first file. In the second, NR runs from 5 to 8 while FNR runs from 1 to 4. For this example, NR == FNR holds only while reading the first file. { array[$1]; next } stores each first-column identifier as a key. next skips the remaining actions for that record. When reading the second file, $0 in array {print} tests whether the whole line matches a stored key and prints matching records.

Max
Mike

2. Replace identifiers in the second file using a mapping from the first

For example, the mapping replaces Mike in file2 with Mike2. Code:

awk -F "\t" ' NR==FNR  { array[$1] = $2; next } $0 in array {$0 = array[$0]} {print} ' file1.txt file2.txt

array[$1] = $2 builds a key–value mapping from the first file. When reading the second file, a matching key is replaced by its mapped value.

Label the duplicate headers in a fasta file

Suppose a FASTA contains duplicate gene identifiers that should receive distinct suffixes. Transform:

>apple
ATGCATGC
>apple
ATCGATCG

into:

>apple.1
ATGCATGC
>apple.2
ATCGATCG

→ Code:

awk '{ if (NR==FNR) {
					if (/^>/){s[$0]++}
					next;}
			 if (/^>/){k[$0]++;
								 if (s[$0]>1) { $0=$0"_"k[$0] }
								} 
print
}'
input.fa input.fa > output.fa


# OR a easier way
awk '/^>/ {if (seen[$0]++){$0=$0"_"seen[$0]} } {print}' input.fa > output.fa
  • s[$0]++ counts the number of times that specific header line appears in the file. For example, given this FASTA:
>header1
>header2
>header1
>header3
>header2

The s array will contain:

s[">header1"] = 2
s[">header2"] = 2
s[">header3"] = 1

Result:

>header1_1
>header2_1
>header1_2
>header3_1
>header2_2
  • if (s[$0]>1) { $0=$0"_"k[$0] } : if a header occurs more than once, append its occurrence number to distinguish the repeated entries.

Naming caveat: if the input already contains suffixes such as xxx_1, choose a different convention before using this code.

Conditional expressions in awk

awk -F ' ' 'NR==FNR{ map[">"$1] = ">"$2; next;} {if (/^>/) {$0=(map[$0]) ? map[$0] : $0} print} ' Hofstenia_miamia/processed/Hof_pep.vs.nucl_output_1.11 \
Hofstenia_miamia/processed/Hof_output_1.2.faa > Hofstenia_miamia/processed/Hof_output_1.12.faa

The ternary conditional expression is: (condition) ? (expression_if_true) : (expression_if_false)

Filter rows by a pattern in a particular column

awk -F"\t" '$2 !~ /^g[0-9].*|Hof/' mmseqs2_RBH/Prae_Hof_RBH_pep.result.sort > mmseqs2_RBH/Prae_Hof_RBH_pep.result.sort.deHofPepID

sed in awk

Apply a substitution to a particular field rather than the entire line.

awk -F"," '{gsub("-","_",$7)}1' OFS="," celltype_markers_Hulett.csv

excape single quote in awk

lvl1    KC-a'/b'        vsRest  WholeBrain      Up      chr3L:21863491-21864316 ""      3.9e-250        4.2e-245        0.84    1       0.51    0.49    KC-a'/b'--lvl1_vsRest_wWB
lvl1    KC-a'/b'        vsRest  WholeBrain      Up      chrX:14125993-14126604  ""      6.9e-271        7.4e-266        0.79    1       0.5     0.5     KC-a'/b'--lvl1_vsRest_wWB

To select records whose second column is KC-a'/b':

awk '$2 == "KC-a'\''/b'\''"'

The example below shows how shell quoting and escaped quote characters interact when a literal apostrophe appears in the awk pattern.

← Back to all posts