Best Of Links Post

Projects:
     * [ GIT Hub ] [ Fedora ]   Livemedia Modifier (LMM)
     * [ GIT Hub ] [ Fedora ]   Fedora ARM Image Installer (FAII)
     * [ GIT Hub ] [ Fedora ]   Mini-Koji Build System (Moji)

Posts:
     * [ GIT Hub ] [ Fedora ]   Koji Follow Python Script
     * [ GIT Hub ] [ Fedora ]   Nagios Koji Perl Script
     * ARM Assembly Intro
     * Seneca College Research Papers

Convert a really big hex string into a decimal string using bash & bc

x=1 ; m=1 ; n=0 ; while true ; do h=`cat /tmp/prime.txt | tr -d ' \t\r\n' | rev | cut -c $x | sed -e 's/A/10/' -e 's/B/11/' -e 's/C/12/' -e 's/D/13/' -e 's/E/14/' -e 's/F/15/'` ; if [ "$h" == "" ] ; then break ; fi ; n=`echo "$n + ($h * $m)" | bc` ; m=`echo "$m * 16" | bc` ; x=`echo "$x + 1" | bc` ; done ; n=`echo "$n" | tr -d ' \t\r\n' | sed -e 's/[^0-9]//g'` ; echo "$n"

Edit: Random self plug – http://fedoraproject.org/wiki/Fedora_ARM_Installer

Just some end-of-semester documentation links

For building and maintaining packages in Fedora:

- https://fedoraproject.org/wiki/Using_Fedora_GIT
- http://fedoraproject.org/wiki/Bodhi

Generating A GUI Windows EXE From A Python Script

# exes: python-2.7-x32, pyqt4-2.7-x32, py2exe-2.7-x32
# run: cd C:\Users\admin\Desktop\fedora-arm-installer
# files: setup.py, MSVCP90.dll
# run: C:\Python27\python.exe setup.py py2exe

import sys,os
from distutils.core import setup
import py2exe

os.system("rd /s /q .\\build")
os.system("rd /s /q .\\dist")
os.system("rd /s /q .\\fedora-arm-installer-0.0.0-0.x32")

a=open("fedora-arm-installer","r")
b=open("fedora-arm-installer-2","w")
for l in a.readlines():
 m = l
 m = m.replace("sys.stderr","#sys.stderr")
 m = m.replace("proglist.append(progcomd)","proglist.append(re.sub('fedora-arm-installer-2.exe$','helper.exe',progcomd))")
 b.write(m)
b.close()
a.close()

os.system("copy fedora-arm-installer-2 helper /y")

setup(console=['helper'], options={"py2exe":{"includes":["sip"]}})
setup(windows=['fedora-arm-installer-2'], options={"py2exe":{"includes":["sip"]}})

os.system("md .\\dist\\data")
os.system("xcopy .\\data .\\dist\\data")
os.system("rename .\\dist fedora-arm-installer-0.0.0-0.x32")

Edit: Updated the setup.py script above to remove the command line window

# exes: python-2.7-x32, pyqt4-2.7-x32, py2exe-2.7-x32
# run: cd C:\Users\admin\Desktop\fedora-arm-installer
# files: setup.py, MSVCP90.dll
# run: C:\Python27\python.exe setup.py py2exe

import sys,os
from distutils.core import setup
import py2exe

os.system("rd /s /q .\\build")
os.system("rd /s /q .\\dist")

a=open("fedora-arm-installer","r")
b=open("fedora-arm-installer-2","w")
for l in a.readlines():
 m = l.replace("sys.stderr","#sys.stderr")
 b.write(m)
b.close()
a.close()

setup(console=['fedora-arm-installer'], options={"py2exe":{"includes":["sip"]}})
#setup(windows=['fedora-arm-installer-2'], options={"py2exe":{"includes":["sip"]}})

os.system("md .\\dist\\data")
os.system("xcopy .\\data .\\dist\\data")

A Really Basic Koji-Shadow Script

This script will follow the general procedure:
* List the latest tagged (non-inherited) packages from the arch being followed
* Download and import any of the non-arch detected packages
* Sort all missing builds by their creation timestamp and order them to be qued
* Loop thru the missing package list and download and que each build

A simple server-side script to help reduce the size of the Koji sessions table:

#!/bin/bash
pd=$(echo $(date "+%s") - 1800 | bc) ; echo "$pd" ; psql koji -c "delete from sessions where extract(epoch from update_time) < $pd and master != 0;"
while true
do
	pd=$(echo $(date "+%s") - 1800 | bc) ; echo "$pd" ; psql koji -c "delete from sessions where expired = 't' and extract(epoch from update_time) < $pd;"
	sleep 2400
done

A simple script to re-que the latest failed build tasks in case of a build-root breakage:

#!/bin/bash
let x="0"
let l="50 * $1"
while [ $x -lt $l ]
do
	echo "[$x]"
	for t in `curl -sL "$1/koji/tasks?start=${x}&state=failed&view=flat&method=build&order=-completion_time" | grep -Eiv "pcre|libssh|glib2|glibc" | grep -i 'taskinfo.taskID=[0-9]*' | sed -e 's/^.*taskinfo.taskID=\([0-9]*\).*$/\1/g'`
	do
		echo "    [$t]"
		koji -s "$1/kojihub" resubmit --nowait "$t"
	done
	let x="$x + 50"
done

Source Code (Git Hub)
Source Code (Fedora People)

Syncing iTunes Playlists From Windows 7 To A Nexus 4

* Connect both devices to a common WiFi AP and determine their IP addresses (Windows/Nexus)
* Download a capable SSH/SFTP/SCP server app that also allows you to setup a capable login user (Nexus)
* Download and run “dokan-sshfs” and then connect to and mount the phone as a “U:\” drive (Windows)
* Run the “syncplay.js” wscript below to sync a smart playlist called “recent” to the phone (Windows)

var iTunesApp = WScript.CreateObject("iTunes.Application");
var mainLibrary = iTunesApp.LibrarySource;
var playlists = mainLibrary.Playlists;
var numPlaylists = playlists.Count;
var fileSys = WScript.CreateObject("Scripting.FileSystemObject");

while (numPlaylists != 0)
{
	var currPlaylist = playlists.Item(numPlaylists);
	
	if (currPlaylist.Name == "recent")
	{
		var playlistTracks = currPlaylist.Tracks;
		var numTracks = playlistTracks.Count;
		
		WScript.Echo("[" + currPlaylist.Name + ":" + numTracks + "]");
		
		while (numTracks != 0)
		{
			var sourceTrack = playlistTracks.Item(numTracks);
			var targetTrack = ("U:\\sdcard\\Music\\" + fileSys.GetBaseName(sourceTrack.Location) + "." + fileSys.GetExtensionName(sourceTrack.Location));
			var sourceSize = fileSys.GetFile(sourceTrack.Location).Size;
			var targetSize = 0;
			
			if (fileSys.FileExists(targetTrack))
			{
				targetSize = fileSys.GetFile(targetTrack).Size;
			}
			
			//WScript.Echo("[" + sourceTrack.Name + ":" + sourceTrack.Location + ":" + targetTrack + "]");
			
			if (sourceSize != targetSize)
			{
				fileSys.CopyFile(sourceTrack.Location, targetTrack);
			}
			
			numTracks--;
		}
		
		//add check to remove uneeded songs
		//create a playlist xml file
		//force device refresh
		WScript.Echo("Done");
	}
	
	numPlaylists--;
}

2012 in review

The WordPress.com stats helper monkeys prepared a 2012 annual report for this blog.

Here’s an excerpt:

600 people reached the top of Mt. Everest in 2012. This blog got about 2,900 views in 2012. If every person who reached the top of Mt. Everest viewed this blog, it would have taken 5 years to get that many views.

Click here to see the complete report.

A new Koji instance for a new arch

So we’ve fired up a new Koji setup to facilitate our future builds of ARMv6hl capable packages. The new site can be reached and tracked here ( http://japan.proximity.on.ca/koji/ ). Right now, it is attempting to catch up by “cleanly” and natively building Fedora Linux 17 for ARMv6hl for the first time. In the future, we may run the koji-shadow build script which matches the build process of the secondary && primary arch’s and employs a smarter que’ing technique.

Note:

For a new arch, you'll have to patch Koji as well. For example, you can find any references to armhfp/armv7hl and so on and replace them with armv6hl using this command:

rpm -qa | grep -Ei '(rpm|yum|koji)' | while read p ; do echo "$p" ; rpm -ql "$p" | xargs grep -i "armhfp" 2> /dev/null ; done

This (brute-force) build script can be run concurrently with other copies of itself at the same time.

que.sh

#!/bin/bash
while true
do
	x=1
	last="a"
	pkgn="b"
	tskl=`curl -sL 'http://japan.proximity.on.ca/koji/tasks?view=flat&state=active&method=build&order=-id' | grep -i 'through' | sed -e 's/^.*through [0-9]* of //g' -e 's/<.*$//g' | head -n 1`
	if [ "$tskl" == "" ]
	then
		tskl=0
	fi
	while [ $tskl -lt $1 ]
	do
		pkgn=`head -n "$x" pkgs.txt | tail -n 1`
		if [ "$pkgn" == "$last" ]
		then
			exit
		fi
		#echo "$pkgn"
		if [ ! -f "$pkgn.src.rpm" ]
		then
			echo "    building [$pkgn]...."
			koji -s 'http://arm.koji.fedoraproject.org/kojihub' download-build --arch=src --topurl='http://arm.koji.fedoraproject.org/' "$pkgn"
			koji -s 'http://japan.proximity.on.ca/kojihub' build f17 "$pkgn.src.rpm" --nowait
			let tskl="$tskl + 1"
		fi
		last="$pkgn"
		let x="$x + 1"
	done
	sleep 30
done

This script below will reset the failed build attempts after each run is completed.

rst.sh

#!/bin/bash
rm -fv *.src.rpm
for tagd in `koji -s 'http://japan.proximity.on.ca/kojihub' list-tagged f17 --quiet --latest --inherit | awk '{ print $1 }'`
do
	touch "${tagd}.src.rpm"
done

And since Koji seems to suck (no offence) at managing its sessions table, this script will clear out any expired sessions which are more than 10 minutes old so the table doesn’t become huge and slow.

exp.sh

#!/bin/bash
while true
do
        psql koji -c "select update_time from sessions where expired = 't';" | cat | grep -i '[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9] [0-9][0-9]:[0-9][0-9]:[0-9][0-9]' | while read date
        do
                a=`date --date="$date" "+%s"`
                b=`date "+%s"`
                let c="($b - $a) / (60 * 10)"
                if [ $c -gt 0 ]
                then
                        echo "deleting [$date] exp [$c] from [`date`]"
                        psql koji -c "delete from sessions where expired = 't' and update_time = '$date';"
                fi
        done
        sleep 300
done

A lesson in how / why companies hire & fire

Was doing some late-night Slashdot reading and came across an article submission which had a blog post about a facebook employee who had been fired. Although it seems like personal rant, there are some interesting points which give insight on how companies think about employees.

http://okdork.com/2012/09/29/why-i-got-fired-from-facebook-a-100-million-dollar-lesson/

Quoting:

“When you hire people there are three types of employees:

1- Grower. Someone who starts when the company is small and improves / adapts their skills as the company scales.

2- Show-er. Someone who can be good for the company where they are now but NOT where they are going.

3- Veteran. They’ve done it before and it’s second nature for them to teach you how to do it in your company.”

…

“Firing:

1- It stings the person WAY more than the company. I thought every day that the company missed me but I’ve learned they just keep going on with business. AND (UN)FORTUNATELY most businesses get better. So be stern when letting someone go but be reasonable and thoughtful to how it must feel. I encourage everyone to get fired once so they know that feeling. It’s unbelievable and something to definitely learn from.

2- EVERYONE is replaceable. You are NOT special and there is guaranteed someone better than you on this planet. So be the opposite, find the way to be invaluable where you work. This doesn’t mean locking things into you but opening things up so you are trusted and subsequently valued more.

3- Most people when they get let go, they know it’s time. They may not want to accept what their subconscious tells them but they know it’s right and it opens them up to something better. Instead of throwing them away, help guide / work with them to see what is their true calling and better suited for them.”

edit

I also came across another article which highlighted a video about motivation which I had posted before on this blog. It tries to explain why we do what we do in this open source world: http://www.youtube.com/watch?v=u6XAPnuFjJc ( http://www.ted.com/talks/dan_pink_on_motivation.html )

Bootstrapping Fedora ARM For A New Arch (v6hl)

Note: These steps assume you have a gcc compiler capable of producing the correct assembly instructions for the intended new arch

Step 1: Add the necessary architecture definitions

/usr/lib/rpm/platform/armv6hl-linux/macros

...
%optflags               -O2 -g -march=armv6 -mfloat-abi=hard -mfpu=vfp
%__isa_name             armv6hl
...

/usr/lib/rpm/{,redhat}/{rpmrc,macros}

...
optflags: armv6hl -O2 -g -march=armv6 -mfloat-abi=hard -mfpu=vfp
arch_canon:     armv6hl: armv6hl        12
buildarchtranslate: armv6hl: armv6hl
arch_compat: armv6hl: armv7hnl
buildarch_compat: armv6hl: armv7hnl
...
%arm    armv3l armv4b armv4l armv4tl armv5tel armv5tejl armv6l armv6hl armv7l armv7hl armv7hnl
...
optflags: armv6hl %{__global_cflags} -march=armv6 -mfloat-abi=hard -mfpu=vfp
buildarchtranslate: armv6hl: armv7hnl
...

/usr/lib/python2.7/site-packages/rpmUtils/arch.py

...
"armv6hl": "armv7hnl",
...

Step 2: Compile the latest version of gcc for your related arch using rpmbuild (--target armv6hl)

Step 3: Modify mock to hijack && inject these new files into all future build roots

/usr/sbin/mock

...
def do_rebuild(config_opts, chroot, srpms):
...
    chroot.init()

    ldst = (chroot.basedir+"/root"+"/usr/lib/rpm/platform")
    import shutil
    shutil.copytree("/usr/lib/rpm/platform/armv6hl-linux",ldst+"/armv6hl-linux")
    os.system("for f in /usr/lib/rpm/{,redhat}/{rpmrc,macros} ; do cp -fv \"$f\" \""+chroot.basedir+"/root/$f\" ; done")
    lcmd = ("/bin/rpm --force --ignorearch --nodeps --root "+chroot.basedir+"/root/ -i /repo/gcc-4.7.0-5.fc17/*.rpm")
    os.system(lcmd)

    chroot.build(srpm, timeout=config_opts['rpmbuild_timeout'])
...

Step 4: Build glibc and all other needed packages using mock (use a helper shell script loop && make sure to patch rpm/yum/redhat-rpm-config later)

/etc/mock/f17v6.cfg

...
config_opts['target_arch'] = 'armv6hl'
...

Step 4.5.0: rpm C code (unified diff patch)

rpm-x.x/lib/rpmrc.c

...
+#      if defined(__linux__) && defined(__arm__)
+       {
+           if (strcmp(un.machine, "armv7l") == 0 ) {
+               if (has_neon() && has_hfp())
+                    strcpy(un.machine, "armv7hnl");
+                else if (has_hfp())
+                    strcpy(un.machine, "armv7hl");
+           }
+           if (strcmp(un.machine, "armv6l") == 0 ) {
+               if (has_neon() && has_hfp())
+                    strcpy(un.machine, "armv6hnl");
+                else if (has_hfp())
+                    strcpy(un.machine, "armv6hl");
+               strcpy(un.machine, "armv6hl");
+           }
+       }
+#      endif   /* arm*-linux */
+
...

Step 4.5.1: yum Python script (mod)

yum-x.x/{yum/misc.py,rpmUtils/arch.py}

...
os.uname().replace("armv6l","armv6hl").replace("armv7l","armv6hl").replace("armv7hl","armv6hl")
...

Step 5: Install a core set of packages needed for a minimal boot (use a capable host/arch)

compose.sh

#!/bin/bash

if [ "$repo" == "" -o "$root" == "" ] ; then exit 1 ; fi

setenforce 0

for pkgn in 'acl' 'attr' 'audit' 'authconfig' 'basesystem' 'bash' 'binutil' 'bzip' 'ca-certificate' 'chkconfig' 'cpio' 'coreutil' 'cracklib' 'cryptsetup' 'curl' '.*curses' 'cyrus-sasl' 'db4' 'dbus' 'dbus-glib' 'diffutil' 'dracut' 'e2fsprog' 'elfutil' 'expat' 'fedora-release' '/file' 'findutil' 'fipscheck' 'gamin' 'gawk' 'gcc' 'gdbm' 'generic-release' 'glib' 'gmp' 'gnupg2' 'gpgme' 'grep' 'gzip' 'hardlink' 'hostname' 'hwdata' 'initscript' 'iproute' 'iptable' 'iputil' '.*kernel' 'keyutil' 'kmod' 'krb5' 'libassuan' 'libcap.*' 'libdb' 'libedit' 'libffi' 'libgcrypt' 'libgpg-error' 'libidn' 'libpwquality' 'libse' 'libssh2' 'libus' 'libut' 'libxml2' 'linux-atm' 'logrotate' 'lua' 'lvm2' 'mingetty' 'net-tool' 'newt' 'nspr' '/nss.*' '/openss' 'openldap' 'pam' 'passwd' 'pcre' 'pinentry' 'pkgconfig' 'popt' 'procps' 'psmisc' 'pth' 'pygpgme' 'python' 'python-chardet' 'python-iniparse' 'python-kitchen' 'python-pycurl' 'python-urlgrabber' 'readline' '/rpm' 'sed' '/setup' 'shadow-util' 'shared-mime-info' 'slang' 'sqlite' 'syslog' 'systemd' 'sysvinit' 'tcp_wrapper' 'texinfo' 'tzdata' 'udev' 'ustr' 'util-linux' 'xz' 'yum' 'yum-metadata-parser' 'yum-util' 'zlib' ; do fill=`find $repo -type f | grep -i "/[^/-]*$pkgn[^/-]*-[^/-]*-[^/-]*/[^/]*.rpm$" | grep -Eiv '(-debuginfo-|.src.rpm)'` ; if [ "$fill" == "" ] ; then echo "FOF $pkgn" 1>&2 ; continue ; fi ; echo "$fill" ; done | sort | uniq | grep -Eiv '/(audispd-plugins|audit-libs-devel|audit-libs-static|cmirror|cpp|cryptsetup-devel|cryptsetup-python|cyrus-sasl-sql|dbus-glib-devel|dracut-fips|emacs-libidn|gcc|glib2-devel|glib2-static|glibc-devel|glibc-headers|glibc-static|glibc-utils|gnupg2-smime|krb5-server|libcap-ng-devel|libgcrypt-devel|libgcj|libgfortran|libgudev1-devel|libitm|libmudflap|libselinux-python|libsemanage-python|libuser-devel|linux-atm-libs-devel|lvm2-cluster|newt-devel|newt-static|openldap-devel|openldap-servers|openldap-servers-sql|openssh-askpass|openssh-ldap|openssl-devel|openssl-perl|openssl-static|pam_|passwdqc|policycoreutils-python|policycoreutils-sandbox|python[0-9]*-debug|python[0-9]*-devel|python[0-9]*-test|python[0-9]*-tkinter|python[0-9]*-tools|rpm-build-[0-9]|rpm-cron|rpm-devel|rpm-sign|rpmdevtools|rpmlint|rsyslog-gnutls|rsyslog-libdbi|rsyslog-mysql|rsyslog-pgsql|rsyslog-relp|rsyslog-snmp|rsyslog-udpspoof|setuptool|slang-slsh|sqlite-tcl|systemd-analyze|texinfo-tex|tkinter|yum-cron|yum-plugin-changelog|yum-plugin-local|yum-plugin-puppetverify|yum-plugin-refresh-updatesd|yum-plugin-tmprepo)[^/-]*-[^/]*$' > pkglist.txt

#for p in $(rpm --force --ignorearch --root $root/ -i $(cat pkglist.txt | grep -Eiv '(gtk|gui|qt|x11)') 2>&1 | awk '{ print $1","$NF }' | sort | uniq) ; do a=$(echo "$p" | cut -d ',' -f 1) ; b=$(echo "$p" | cut -d ',' -f 2) ; q=$(repoquery --whatprovides "$a" | head -n 1) ; r=$(repoquery -qi "$q" | grep -i '^source') ; echo "[$b] -> [$a] -> [$q] -> [$r]" ; done
rpm --force --ignorearch --nodeps --root $root/ -i $(cat pkglist.txt | grep -Eiv '(gtk|gui|qt|x11)') x2

rm -frv $root/usr/share/*doc*

cat << EOF > $root/etc/fstab
LABEL="rootfs"          /                       ext4    defaults,noatime                 1 1
LABEL="boot"            /boot                   vfat    noauto,comment=systemd.automount 1 2
EOF

cat << EOF > $root/etc/sysconfig/network
HOSTNAME=raspiv6hl.local
EOF

cat << EOF > $root/etc/resolv.conf
nameserver 4.2.2.1
EOF

echo 'utmp:x:22:' >> $root/etc/group
echo 'PS1="[\u@\h: \w]# "' > $root/root/.bashrc
chmod 755 $root/root/.bashrc
ln -s .bashrc $root/root/.profile

#raspberrypi
sed -i 's|root:.*:|root:$6$KW0GGbE5$zlEB9.PbHVh8kmXj1WMFGLJGwwthhU4oXn2oNxHZllbUSzTsVhTZ9jts8RC7uicuUCWyrsZ1e2yEj4ErDLOHQ/:15525:0:99999:7:::|' $root/etc/shadow

rm -fv $root/etc/systemd/system/default.target && ln -s /usr/lib/systemd/system/multi-user.target $root/etc/systemd/system/default.target

Step 6: Boot the new Fedora ARM Linux system

The Start Of A New Term With Some Great People

I haven’t blogged in a while because it has taken literally weeks upon weeks of trying to bootstrap ARMv6HL for Fedora, pretty much by myself (plus hints from my boss Chris Tyler). I am not a programmer! But anyway, I think it’s coming along to some degree at least (remind me to always respect the complexity of gcc && glibc). In addition to this, I am planning on working with Seneca college for another year and also, this is officially the first time I’ve hit the gym!!! My friend and co-worker Andrew Greene, who has been working hard on various tasks, decided to show me around the machines today which was pretty cool and I already feel a bit better for myself after having gone (hopefully I’ll keep it up). I also have thank my co-workers Jordan and Paul for encouraging me to go in the first place, along with the other life advice they’ve given me. In addition, we got a new team member named Dmitry who’s been pretty much looking after our build farm and making sure all the machines remain up for all the remote users who depend on its availability.

And on a last note, my friend Leopold Antonio Martinez (who’s been like my best friend for life) and I are planning on looking for places to live. He’s helped me out constantly in the past and been there for me all the time. This next step should help me separate from my Italian parents and gain some independence! Thanks Lee

Anyway, enough of my ranting, here’s hoping for another productive year!

Johnny S.