What are processes & Arguments/Flags?

• Processes are programs on a running machine
• Every program accepts a series of arguments, known as flags
• These flags vary based on the program
• Arguments are generally in the form of `--longopt` or `-a` for single options

Flag

are single character options like `-a`, and don't take an argument

Arguments

Arguments are what you actually pass to the program, and may take an argument, such as `--max-workers 5`.

An Introduction To Files

Files are streams of bytes. Possibly bi-directional, but usually opened for either reading or writing.

A stream has the following valid operations on it.

hasNext

Is there more information available in the file?

read

Get the next value in the stream.

write

Write a value into the stream.

Unix/Linux is all about files. Just about everything is accessible as a file.

Standard Input & Standard Output & Standard Error

By default a program has 3 files open.

Standard Input (stdin)

This is usually connected to the keyboard, so a user can interact with the program. Alternately it can be attached to a file.

Standard Output (stdout)

The default place where print statements go

Standard Error (stderr)

This is where errors are usually written. It's kept separate so you might see the errors when writing a pipeline.

Creating a temporary file

Ever need a file for a small task, but don't want to go to the effort of coming up with a good name. Temporary files are a life saver. They don't stick around, and are guaranteed to have a unique name

How to create a temporary file

MY_TEMP_FILE=$(mktemp)
cat > $MY_TEMP_FILE

A subshell captures what is printed by `mktemp`, and assigned to `MY_TEMP_FILE`. You can write to the file by using `cat` and a redirection. Close the file with CTRL+D

Where am I and how do I move around?

These are some common commands to figure out where you are and what you want to do.

ls

List files in my current working directory

cd

Change directory

pwd

Print working directory (where exactly am I?)

Some Basic Commands

• cut
• awk
• sort & uniq
• cat
• grep
• find
• curl
• xargs

Cut: Remove sections from each line of files

This utility can split sections of a line by bytes or delimiter. It's most useful with simple CSVs. For example, the first and third columns are needed from a CSV

cut -d, -f1,3 random.csv

For more information read the man page.

Awk: pattern scanning and processing language

Awk is an entire language, but most people use it for line oriented processing. It's a fancy version of cut for a lot of purposes. It uses regex matching to take actions and has various blocks to take actions. The default field separator is white space.

# Print lines where the http status code is 404 not found
awk '$9 == "404"'

# Count up status codes
awk '{s[$9]+=1}END{for(c in s)print c " " s[c]}'

# Sum the 3rd column in a csv without headers
awk -F, '{s+=$3}END{print s}'

Generally you won't see too much awk used in these pipelines, as it takes longer to write and digest AWK over a few simple pipeline patterns. Just know it's very powerful, and can do a lot with very few characters.

Sort & Uniq

`sort` sorts lines in a file. `uniq` removes duplicate lines that appear together.

Sort Examples

# Sort by the natural ordering
sort my_words
# Sort by numbers
sort -n my_item_counts
# Sort in reverse ordering
sort -r my_words

Uniq Examples

# Remove duplicates
uniq my_duplicated_words
# find cases of duplicated words, and count them up
uniq -c my_document_broken_to_words_per_line

Cat - concatenate files

Cat takes a series of files to print the contents of

Print a single file

cat foo

Print many files

cat foo bar baz

Grep: Find it

Grep stands for globally search for a regular expression and print matching lines. Commands are generally in the following form:

grep [-P] pattern [files...]

Grep is often used in log processing, like counting up how many times a page or how many times a POST request occurred was viewed in Apache logs.

grep --count POST logs/access_log logs/access_log.old
logs/access_log:8
logs/access_log.old:19

Find: Search for files in a directory hierarchy

Find accepts a series of starting points to traverse the directory hierarchy, and supports various filtering operations. Examples:

# Find all directories below the current directory
find . -type d

# Find only files ending in .txt below the current directory
find . -type f -name '*.txt'

# Find all java files in the test directory that maven will consider a test
find */src/test -name '*Test.java'

These are some common operations, but it supports many more advanced operations. It supports filtering by access time, modified time, permissions, and many other attributes.

Curl : Download A File

Curl is one of the most commonly used programs ever. It makes sending web requests on a ton of protocols super easy. It's great for making API calls or downloading web pages. The result of the request is written to standard out.

Curl supports many flags, but it's a large topic. Generally curl is used like so:

curl $my_url > result.txt

xargs: Use Input to Build A Command Line

Many times we need to select a subset of files and perform an operation on them. Possibly in parallel.

find . -name '*.txt' | xargs zip MyTextFiles.zip

This adds all .txt in your directory hierarchy to a zip file.

Pipelines

Pipes allow commands to be composed and feed into one operation after another. You saw one with the xargs example. To link one command's output to another's standard input, the commands are separated with a pipe(|) character.

# Find a file with the most lines.
find . -type f | xargs wc -l | sort -n | tail -n 2 | head -n 1

# Sort some input
foo_pipeline | sort

# Dump intermediate results to a file.
foo_pipeline_step1 | tee -a $(mktemp) | foo_pipeline_remaining