Counting and sorting

How do you count lines, words and characters?

wc stands for word count, and it counts three things: lines, words and characters (strictly, bytes). Most of the time you want lines, because so much data is one record per line: one bird per line in a tally, one order per line in a CSV, one visitor per line in a web log.

wc -l tally.txt prints the line count followed by the file name. Fed from a pipe, it prints the number alone, because it has no file name to show. That makes grep ... | wc -l the standard way to ask "how many lines match".

wc -w counts words (anything separated by spaces) and wc -c counts bytes. Running wc with no options prints all three. On a real machine the numbers are padded with spaces to line up; the practice terminal prints them with single spaces.

wc -l tally.txt
22 tally.txt

grep heron tally.txt | wc -l
7

Why does wc -l sometimes count one short?

wc -l counts newline characters, not lines you can see. A file whose last line has no newline after it counts one fewer than you expect. Files saved by some spreadsheet programs and Windows editors end this way.

If a count looks one short, check the end of the file with tail -n 1 file | wc -c, or open it in an editor and press End on the last line. A CSV file has another trap: its header row is a line too, so wc -l orders.csv is one more than the number of orders.

How does sort order lines?

sort prints the lines of a file in order, alphabetical by default. It never changes the file; it prints a sorted copy. Add -r to reverse the order.

Default sorting compares text character by character. That's right for names and wrong for numbers: 12.00 sorts before 2.20 because the character 1 comes before 2. Add -n (numeric) whenever the thing you're sorting by is a number. -nr puts the biggest number first.

On a real machine the exact alphabetical order depends on your language settings, called the locale. Capitals and punctuation can land in different places on different computers. If you need the same order everywhere, real scripts often run LC_ALL=C sort.

sort -u tally.txt
blackbird
goldfinch
heron
kingfisher
robin
wren

How do you sort by one column with -t and -k?

A CSV file has several values per line, separated by commas. -t, tells sort that a comma separates the fields, and -k2 says to sort by the second field. Put them together with -n to sort the bakery's prices by cost:

Notice the header line item,price comes first. Its second field is the word price, which counts as zero when read as a number. To keep the header out, start with tail -n +2 prices.csv | sort -t, -k2 -n.

Without -t, sort splits fields at spaces instead, which suits files like wc output or space-separated logs. On a real machine, -k2 means "from field 2 to the end of the line"; write -k2,2 to mean field 2 only. With one column after the separator, as here, there's no difference.

sort -t, -k2 -n prices.csv
item,price
bun,2.20
lemon drizzle,2.60
rye,3.80
sourdough,4.50
slice,9.50
baguette tray,12.00

What does uniq do, and why must sort come first?

uniq collapses repeated lines into one. With -c it also prints how many times each line appeared. The catch is that it only compares each line with the one just before it. Repeats that aren't next to each other are treated as separate runs.

In the tally, herons are scattered through the file. Run uniq -c on it directly and you get a long list of ones and twos, with heron and robin appearing several times. Sort the file first and every heron sits together, so uniq can count them in one go:

This is the one rule to remember about uniq: almost always put sort in front of it. uniq -d shows only lines that appear more than once, and uniq -u only lines that appear once, and both need sorted input for the same reason.

uniq -c tally.txt | head -n 4
      1 heron
      1 robin
      1 heron
      1 blackbird

sort tally.txt | uniq -c
      4 blackbird
      1 goldfinch
      7 heron
      2 kingfisher
      5 robin
      3 wren

How do you get a top three?

Chain the pieces together: sort to group repeats, uniq -c to count them, sort -nr to rank the counts biggest first, and head -n 3 to keep three. This four-stage pipeline answers a surprising number of real questions.

Swap the input and it still works. Which pages of a website were requested most? Which items sold best today? Which email addresses appear on the mailing list twice? Each is a list with repeats, and the same pipeline ranks them.

When two counts tie, their order depends on how sort breaks ties, so a top three can change if you add data. Check the line just below the cut-off with head -n 4 before announcing a winner.

sort tally.txt | uniq -c | sort -nr | head -n 3
      7 heron
      5 robin
      4 blackbird

How do you count different values?

To count how many different species were seen, collapse the repeats and count the lines left: sort tally.txt | uniq | wc -l prints 6. sort -u does the sort and the de-duplication in one step, so sort -u tally.txt | wc -l gives the same answer.

Leave out the sort and the number is too high, because each separated run of the same word counts as another value. The result looks plausible, which is what makes this mistake easy to miss.

A quick sanity check helps: the number of different values can never be larger than the total from wc -l, and if it is close to the total, look at the list with sort -u to see whether near-duplicates such as Heron and heron are being counted apart.

What do AI tools get wrong when counting?

The most common slip is uniq -c without a sort in front, which produces counts that look reasonable and are wrong. Another is sort -r on numbers without -n, so 9 ranks above 12.

They also forget header rows, which add one to every wc -l and put a stray line into sorted output, and they write sort -k2 for a CSV without -t,. Before trusting a count, check it a second way, such as grep -c heron tally.txt for one species.

Field notes

• wc -l counts lines, which often means records.
• sort puts lines in order; add -n for numbers and -r to reverse.
• uniq -c counts repeats, but only next to each other, so sort first.
• sort | uniq -c | sort -nr | head -n 3 gives a top three.

Easy to get wrong

uniq only merges repeats that are next to each other. Without sorting first, the same species shows up several times with small counts.

Without -n, numbers are compared as text, so 12.00 sorts before 2.20.

Text order happens to work while every count is a single digit. Once a count reaches 10, it sorts below 9 unless you add -n.

The shell empties the file before sort reads it, here and on a real machine, so you end up with an empty file. Write to a new file, as the next lesson explains.

Questions people ask

Does sort change my file?

No. It prints a sorted copy. To keep the result, save it to a new file with >, covered in the next lesson.

Why do capital letters sort strangely?

Sorting rules come from your locale. Some put all capitals before lowercase; others mix them. Add -f to ignore case, or lowercase the data first.

What is the difference between sort -u and uniq?

sort -u sorts and removes duplicates in one step. uniq only removes neighbouring duplicates, but with -c, -d or -u it can count or pick out repeats, which sort -u can't.

How do I count matches without wc?

grep -c pattern file counts matching lines directly. Use wc -l when the lines come out of a longer pipeline.