For the last three days I have been in Puppet training (April 28th – May 1st), which rocked, completely. The trainer Hunter Haugen was awesome, not only is he extremely knowledgeable (just don’t throw down the W word) but thoughtful, honest, and took the time to listen to our pain points and give some good advice (you can read his blog here for fantastic tips on vim). I am not going to bore you with the details of the training, other than to say if you’re considering using Puppet in your environment, shell out for the training, it is definitely worth it.

Q: Rename all .html files in dir to .htm

A: Obviously there are quite a few ways to do this, here are just a couple.

So for testing purposes we'll create a directory in /tmp, say test and cd there:

prompt:> mkdir /tmp/test
prompt:> cd !$

Now we'll create a few temporary html files to rename.

prompt:> for i in {1..10}; do touch $i.html; done

Now to rename them en masse:
First way (courtesy of Basketcase):

prompt:> for i in $(ls *html); do mv $i $(echo $i | sed 's/\(.*\)html/\1htm/'); done

Second way (courtesy of myself):

prompt:> for i in $(ls *html); do mv $i $(basename $i .html).htm; done

First way breakdown:
	1. loop through all the specified files, *html in our case.
	2. here is where the magic with sed happens, echo the filename $i, then use sed to greedy match our pattern and substitute it with the new pattern. We use the first part of the matching pattern with "\1" (which will expand to 1. 2. 3., etc) and then replace the last part of the pattern. As an example of this try in the working directory "for i in $(ls); do echo $i | sed 's/\(.*\)htm/\1/'; done" without doing the replacement, it should print all the files and the ones that match should be printed as "filename.", so in our dir it should look something like 1. 2. 3. 4. 5., etc.

Second way breakdown:
	1. loop through all the specified files, *html in our case.
	2. use basename utility to strip off the first pattern (.html) as we move the file to .htm. 

So I created 10k files 1-10000.html and timed the following two commands, this is by no means scientific, I was just curious what the timing was on each of them with more than 10 files.

On 10k files with first method:

prompt:> time for i in $(ls); do mv $i $(echo $i | sed 's/\(.*\).html/\1htm/'); done
 
real	1m36.604s
user	0m15.842s
sys	0m58.361s

On 10k files with second method:

prompt:> time for i in $(ls); do mv $i $(basename $i .htm).html; done
real	1m4.737s
user	0m9.260s
sys	0m38.631s

First way, with backtick `` interpolation:

prompt:> time for i in `ls *html`; do mv $i `echo $i | sed 's/\(.*\)html/\1htm/'`; done
 
real	1m18.076s
user	0m15.478s
sys	0m56.438s

Addendum:
Basketcase also pointed out that using perl rename is much faster, though since I am on an OSX box, it is not native and hence why I personally chose not to go that route.

linux prompt:> time rename  's/\.html/\.htm/' *.html
 
real    0m0.270s
user    0m0.204s
sys    0m0.064s

tl;dr – Learning Haskell and functional programming, yay!

So I have finally decided after years of using imperative style languages that I am going to Learn me some Haskell for a greater good, as I have been curious about the functional side of programming. Now it has taken me quite some time to get here, I have been toying with the idea of functional programming for quite a while, just haven’t gotten around to it as the imperative languages seem to be just fine for what I do for a living, Systems Administration/Engineering. I have been toying with either learning Haskell or Erlang, and have decided on Haskell. Why?, well I don’t really know just kind of happened that I chose Haskell.

About a day after I started reading Learn you a Haskell for a Great Good! I noticed this article – (The Downfall of Imperative Programming) fly across my twitter stream from @LordCope. Now, it is a well written article and surprisingly the comments are for the most part well thought out (I really abhor reading commentary on the internets), and I have to agree with what Bartosz says here in response to a comment about functional programming being the downfall of the imperative languages, “Of course it’s an exaggeration :-) . Just like object oriented programming didn’t eliminate C, Haskell will not eliminate Java. It will just dislodge it into its well deserved niche.” I personally, and honestly think that to be the best that one can be it is sometimes necessary to open one’s mind and try new things, and for me right now that is Haskell and functional programming, and so far I am really enjoying myself.

After I finish with Learn you a Haskell for Great Good! I am going to move on to Real World Haskell which is also freely available online, and while I am not associated with any of the authors in any way, shape or form, if you like the books I strongly encourage you to purchase them as they require a lot of effort to write.

So sitting here on the train I got to thinking about some of the standard aliases that I use, or add to a system upon first installation and use.

This does not stave off the infamous “rm -rf – pebkac error” but to keep myself from general stupidity I tend alias rm, cp and mv first and foremost (that way for the most part if I make a bad decision at least I was prompted to):

alias rm='rm -i'
alias cp='cp -i'
alias mv='mv -i'

Also, back in the day, when I started messing with Linux, I loved the terminal colors, now not so much. So I alias away the colors, make them appear in columns and with characters to denote the type of file (‘/’ for directory, ‘*’ for executable, etc.).

alias ls='ls -CF'

Because “maths are teh hard” I tend to alias df so that it appears in human readable format. Sometimes I’ll alias free, but not always – sometimes I like to live dangerously. Conversely since I tend to work mainly on OSX and free isn’t a built in (yes I know it can be added with ports), I tend to just forget about it.

At the end I will then setup personal aliases that are meaningful to me, but may not have any meaning to someone not living inside my head. For instance if there are a few hosts that I know that I will be sshing to quite frequently, like say jump hosts, I will alias them down to singular commands, usually the hosts name to make getting there quicker and easier.

As with most things in *nix world there are more than one way of doing things. Here is a nifty trick I picked up from a former coworker. Sometimes when I needed to do disk reporting it was handy to have the column header (remember this is my use case) for those who don’t necessarily know what the columns mean when doing a df or similar and grepping for a particular volume / disk (read upper management). I am fairly certain that there are even more ways of accomplishing this, but here are two that I’ve used. Also be aware, that you need to know what you’re looking for, or the pattern you need to match.

Note: this was done from my MBP, so in Linux ymmv*.

sed:

Before:
df -h | grep disk
 
/dev/disk0s2    465Gi  123Gi  342Gi    27%    /
/dev/disk1s2    110Mi   94Mi   16Mi    86%    /Volumes/VirtualBox
/dev/disk2s0s2   25Mi   22Mi  2.7Mi    90%    /Volumes/VZAccess Manager
 
After:
df -h | sed -n '/^Filesystem/p;/disk/p'
 
Filesystem       Size   Used  Avail Capacity  Mounted on
/dev/disk0s2    465Gi  123Gi  342Gi    27%    /
/dev/disk1s2    110Mi   94Mi   16Mi    86%    /Volumes/VirtualBox
/dev/disk2s0s2   25Mi   22Mi  2.7Mi    90%    /Volumes/VZAccess Manager

awk:

Before:
df -h | grep disk
 
/dev/disk0s2    465Gi  123Gi  342Gi    27%    /
/dev/disk1s2    110Mi   94Mi   16Mi    86%    /Volumes/VirtualBox
/dev/disk2s0s2   25Mi   22Mi  2.7Mi    90%    /Volumes/VZAccess Manager
 
After:
df -h | awk '/^Filesystem {print;}; /disk/ {print;}; {next;};'
 
Filesystem       Size   Used  Avail Capacity  Mounted on
/dev/disk0s2    465Gi  123Gi  342Gi    27%    /
/dev/disk1s2    110Mi   94Mi   16Mi    86%    /Volumes/VirtualBox
/dev/disk2s0s2   25Mi   22Mi  2.7Mi    90%    /Volumes/VZAccess Manager

sed breakdown:
With the sed line we are matching any line that begins with Filesystem “/^Filesystem” and printing it “/p”, then matching any line that has disk “/disk” and printing it “/p”. The “-n” option to sed tells it to only print what we are looking for – from the sed manpage on OSX – “-n By default, each line of input is echoed to the standard output after all of the commands have been applied to it. The -n option suppresses this behavior.” If you were to remove the “-n” switch it would look something like this:

 
df -h | sed '/^Filesystem/p;/disk/p'
 
Filesystem       Size   Used  Avail Capacity  Mounted on
Filesystem       Size   Used  Avail Capacity  Mounted on
/dev/disk0s2    465Gi  123Gi  342Gi    27%    /
/dev/disk0s2    465Gi  123Gi  342Gi    27%    /
devfs           199Ki  199Ki    0Bi   100%    /dev
map -hosts        0Bi    0Bi    0Bi   100%    /net
map auto_home     0Bi    0Bi    0Bi   100%    /home
/dev/disk1s2    110Mi   94Mi   16Mi    86%    /Volumes/VirtualBox
/dev/disk1s2    110Mi   94Mi   16Mi    86%    /Volumes/VirtualBox
/dev/disk2s0s2   25Mi   22Mi  2.7Mi    90%    /Volumes/VZAccess Manager
/dev/disk2s0s2   25Mi   22Mi  2.7Mi    90%    /Volumes/VZAccess Manager

printing the matches multiple times due to matching on the “/^Filesystem” and matching on the “/disk”.

awk breakdown:
With the awk one, we are matching the line that begins with “/^Filesystem” and printing it, then matching the line with “/disk/” and printing, then followed by a statement “{next;}” to keep processing and skip unless the pattern matches.

Also while writing the post, thought of another one that it might be useful, though it really depends on what you’re running on your system. Thought of a little netstat example though I will go with the caveat here that netstat can easily make this a moot point, take this with a grain of salt.

Before:
 
netstat -anl | grep -i listen
 
tcp46      0      0  *.9292                 *.*                    LISTEN     
tcp46      0      0  *.9302                 *.*                    LISTEN     
tcp46      0      0  *.9301                 *.*                    LISTEN     
tcp46      0      0  *.9200                 *.*                    LISTEN     
tcp46      0      0  *.9300                 *.*                    LISTEN     
tcp46      0      0  *.62056                *.*                    LISTEN     
tcp4       0      0  *.62056                *.*                    LISTEN     
tcp4       0      0  127.0.0.1.51093        *.*                    LISTEN     
tcp4       0      0  127.0.0.1.26164        *.*                    LISTEN     
tcp4       0      0  *.17500                *.*                    LISTEN     
tcp4       0      0  127.0.0.1.631          *.*                    LISTEN     
tcp6       0      0  ::1.631                                       *.*                                           LISTEN     
ffffff80147a27d0 stream      0      0 ffffff8017e13f00                0                0                0 /tmp/launch-H5oHbg/Listeners
 
After:
 
netstat -anl | sed -n '/^Proto/p;/LISTEN/p'
 
Proto Recv-Q Send-Q  Local Address          Foreign Address        (state)    
tcp46      0      0  *.9292                 *.*                    LISTEN     
tcp46      0      0  *.9302                 *.*                    LISTEN     
tcp46      0      0  *.9301                 *.*                    LISTEN     
tcp46      0      0  *.9200                 *.*                    LISTEN     
tcp46      0      0  *.9300                 *.*                    LISTEN     
tcp46      0      0  *.62056                *.*                    LISTEN     
tcp4       0      0  *.62056                *.*                    LISTEN     
tcp4       0      0  127.0.0.1.51093        *.*                    LISTEN     
tcp4       0      0  127.0.0.1.26164        *.*                    LISTEN     
tcp4       0      0  *.17500                *.*                    LISTEN     
tcp4       0      0  127.0.0.1.631          *.*                    LISTEN     
tcp6       0      0  ::1.631                                       *.*                                           LISTEN

Again, this makes the presumption that you know what you’re looking for, and if you tried to do that with ESTABLISHED in netstat it may be moot unless you narrowed down to less than a screen. Also I’ll leave the awk version and looking for specific ports / port ranges as an exercise for the reader. Also astute readers will notice that we lost the last output from the first netstat command due to not grepping indiscriminately.

I hope this information comes in handy, share and enjoy!

*For those that may not know, ymmv = Your Mileage May Vary.

I know that this is general knowledge, but putting it down here so I remember where I put it. So to create multiple directories with the structure ~/projects/code/ I would use:

localhost: mkdir -p ~/projects/code/{c,perl,python,ruby,shell}

Got the reference for it here. *I always forget the which is correct “{” vs “[“.

Well hello there. I was planning on putting together some long drawn out post about starting to blog again, but figured what the hell, might as well just start doing it. So, though it has been a while, and a new job has been acquired (main reason behind the lack of postage in the preceding months) that will all be covered in time. I also need to figure out a way to get basketcase to post again, maybe I’ll throw something at him when next I see him. So, I will be back shortly with more braindumpage, and hopefully some useful tidbits in there from time to time.

I originally posted this on my blog http://www.barrymorrison.com/2011/09/04/learning-python-resources/ but thought it’d be a good fit over hear as well. These are simply links to Python resources I’ve collected recently in an effort to tech myself Python. Hopefully others can find it equally valuable.

http://code.google.com/edu/languages/google-python-class/index.html

http://i.imgur.com/YQafj.png

http://code.google.com/appengine/docs/python/gettingstarted/

http://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-00-introduction-to-computer-science-and-programming-fall-2008/

http://diveintopython.org/toc/index.html

http://codingbat.com/

http://www.lightbird.net/py-by-example/

http://rgruet.free.fr/PQR26/PQR2.6.html

http://www.udemy.com/learn-python-the-hard-way/

http://learnpythonthehardway.org/

http://www.swaroopch.com/notes/Python

http://www.pythonchallenge.com/

http://www.youtube.com/user/thenewboston#p/search/0/4Mf0h3HphEA

I was working on a script that parsed log files. The file names were log.mmddyy (log.080811). This was a little bit of a learning experience for me (new to shell scripting). I was working on a Solaris 10 box, but I thought I’d do all my dev/testing on Ubuntu. Big mistake. Linux was running bash/gnu tools. Solaris was not. After I figured out how to do it on Ubuntu, I had to figure out how to do it on Solaris (thanks Scott!), then I did it in Powershell (had done something similar already for another script) and since I’m teaching myself Python, I wanted to do it in Python.

Are they the “right” way to do them? They work! Probably not a “right” way to do them, as there is always more than one way to do something

Shell

#!/bin/bash
 
os=`uname`
 
if [ $os == 'Linux' ];
    then
    yesterday="$(date -d 'yesterday' +%m%d%y)"
    echo $yesterday
elif [ $os == 'SunOS' ];
    then
    yesterday=`TZ=GMT+24 date +%m%d%y`
    echo $yesterday
fi

Gnu Date

#!/usr/bin/bash
 
## Get Yesterday
yesterday="$(date -d 'yesterday' +%m%d%y)"

Powershell

$yesterday = (get-date -format mmddyy).adddays(-1)
$yesterday.tostring('MMddyy')

Python

 
import datetime
today = datetime.datetime.today()
one_day = datetime.timedelta(days=1)
yesterday = today - one_day
yesterday.strftime('%m%d%y')

In an earlier post, Scott did this in a simple shell script. I was curious if I could do it in Python since I am learning Python.

Below is my first attempt. Watch for upcoming updated version.

import sys
 
if not sys.argv[1:]:
   mac = raw_input ("Enter MAC address\n")
   print "Mac is %s\n"; % (mac)
   #sys.exit(0)
else:
   mac = sys.argv[1]
   print "Mac is %s\n" % (mac)
 
##I'm checking to see if someone is giving me in the right format
maclen=len(mac)
if maclen == 12:
    ## converting to list, since strings are immuatable
    maclist=list(mac)
    ## defining positions on list
    positions=2,5,8,11,14
 
    for i in positions:
        maclist.insert(i,':')
    ##joining list at the ','
    newmac=",".join(maclist)
    ## stripping ',' out and converting to lowercase.  
    final12mac=newmac.replace(',','').lower()
 
    print final12mac
 
##assuming that 17 characters will either have - or : in MAC address    
elif maclen == 17:
    ##replacing - with : and converting to lowercase
    mac17final=mac.replace('-',':').lower()
    print mac17final
else:
    ## I'm telling them that if it isn't 12 or 17 characters, it has X chars
    print "There are only %s characters" % (maclen)