Manipulating CSV files with Unix Shell

The quick and dirty way to manipulate CSVs from the Shell.

Say you want to create a new CSV subset file with just two specific fields from the original CSV.  The one liner code is:



awk -F\, '{printf "%s,%s\n",$1,$3}' original.csv > new.csv


How about sorting on the second field?


cat new.csv | awk -F\, '{printf "%s,%s\n",$1,$2}'  | sort -t, -k 2 > newsorted.csv


Iterative Perl and Bash search and replace words in multiple files at once

Learning the ropes around applying Perl magic to mundane chores.

-e: tells perl to treat the line that follow as a perl script.

': wrap the line of code in single quotes to prevent perl from attempting to interpret special characters. However \' causes problems and is never printed literally even if escaped with a \ and enclosed in single quotes. I still don't know the workaround to this quirk.

-p: causes Perl to assume the following loop around your program, which makes it iterate over filename arguments.

-pe: p and e combined over a line of code iterates over every file on the command line or on the text fed to it on STDIN

-i: lets you edit files in place.

s/oldstring: Searches for the following old string.

/newstring/g: Does a global replace of the old string with the new string.


So the mass search and replace command would look like


perl -pi.orig -e 's/oldstring/newstring/g' filename

Though this works in a directory, somehow it doesn't iterative work in the sub-directories. So it's time fore bash script.

#!/bin/sh
# tells the interpreter to use the shell for interpreting these following commmands

# Use grep to find the files with the string of interest
for file in $(grep -il "oldstring" *.tex)
# Start bash script
do
# give the file list to sed
# search and replace (same as perl above)
# sed doesn't mess with originals so write the file with changes to a temporary file
sed -e "s/oldstring/newstring/ig" $file > /tmp/tempfile.tmp
# Move the temporary file created to the file list that you created in the grep step
# The new file list will have replaced strings
mv /tmp/tempfile.tmp $file
done


Regular Expressions

This is a super-awesome website that parses any regex you throw at it into plain ole English!

Match the boundaries of a word
Exceedingly useful for inserting quotes and commas before and after words
\<([^ ]*)\>

This will find the space bounding the word:
\<: Start of the word
( ): Can contain a list of options each separated by a "pipe" |
[] Restricts the possible values that the pattern will match in a particular position.
[^*]: Inside the bracket it tells the parser to OMIT the characters listed. Here, omit everything.
* any number of the preceding character are allowed but none are required
\>: End of the word

In the replace box:
\1 This is a back reference to the submatch within the 1st parentheses.
"\1" Tell geany to replace the boundaries of the 1st match with quotes.

Positional Characters:
^: Caret matches START pattern.
$: Dollar symbol ENDING pattern.

Wildcards:
. matches any single character
\.: is a decimal point
* any number of the preceding character are allowed but none are required

Character Classes
[] Restricts the possible values that the pattern will match in a particular position.
[A-Z] Supports ranges of characters within the brackets.

Combo Meta Characters and Character Classes
^[A-Z] Matches any word starting with a capital alphabet.
[0-9] or \d Identifies any word with at least one number between 0 and 9 (or digits) in it.

Omission: Double meaning of Caret
[^0-9] or \D: Inside the bracket it tells the parser to OMIT the characters listed. \D is a non-digit.
^[0-9]: Outside the bracket it tells the parser to match the characters listed at the BEGINNING of the pattern.

The Repetition Indicator
Combine the wildcard and repetition indicator. Examples:
^\D*: Starts with a series of non-digits
\D{2, 6}: Two to six non-digits
\d{5}: Five digits

The Optional Indicator
,?: The character (in this case the comma) preceding the ? may occur one time or it may not.
+,?:  Look for at least one character prior to the comma.

A Range of Options
( ): Can contain a list of options each separated by a "pipe" |

Ignore Case
/i: i after the second forward slash ignore case.

Complicated Expressions

("/(int.*l?|wo?r?ld|glo?b[ae]?l?)/i")
/ start parsing
int.*l?: words starting with int, any single character, in fact any number of single characters are allowed but none are required, i.e, it could just be "int", followed by l, OR word starting with wo, o may occur one time or it may not, r may occur one time or it may not, followed by ld, OR gl followed by,o may occur one time or it may not, b, [anything from a through e] may occur one time or they may not, l may occur one time or it  may not.
/i: ignore case, end parsing

Writing complicated expressions
Start with simple building blocks
Create a variable to match misspelled word headache
he?a?d.?[ -]?(ache)


Concatenate regular expressions with
||
Separate them with .* :patterns separated by any kind of text


Standardizing Spelling
s/: indicateds that the regular expression will be used in a substitution
Regular expression parser will search for the pattern between the first two forward slashes and replace it with the text found between the second and the third forward slashes.