Search notes:

Shell command: uniq

uniq [OPTION]… [INPUT [OUTPUT]]

Options

-c --count Prefix lines by the number of occurrences
-d --repeated Only print duplicate lines, one for each group. This option is very useful to determine if a text contains duplicated lines.
-D Print all duplicate lines
--all-repeated [=METHOD] Like -D, but allow separating groups with an empty line. METHOD is one of none (the default), prepend or separate
-f --skip-fields= N Avoid comparing the first N fields
--group [=METHOD] Show all items, separating groups with an empty line. METHOD is one of separate (the default), prepend, append or both
-i --ignore-case Ignore differences in case when comparing
-s --skip-chars= N Avoid comparing the first N characters
-u --unique Only print unique lines
-z --zero-terminated Line delimiter is NUL, not line break
-w --check-chars= N Compare no more than N characters in lines
--help
--version

Using -d to verify if all texts are unique

The -d options allows to quickly determine if a file contains duplicated lines.
The following pipeline prints nothing:
echo 'This
text
does
not
have
duplicates' | sort | uniq -d
This pipeline prints the word this because it is duplicated:
echo 'in
this
text,
the
word
this
occurs
twice' | sort | uniq -d

Count the frequency of values

The following pipeline counts how many git commits each author has contributed to a git repository:
$ git log --format='%aN' | sort | uniq -c  | sort -nr

Misc

uniq does not detect repeated lines unless they are adjacent.
Try using sort -u (without uniq).

See also

sort, comm, join
Shell commands

Index