Matlab normxcorr2 implemented in python

For some reason there is no direct implementation of normxcorr2 from Matlab or Octave in numpy or scipy. This is a near exact normxcorr2 taken from Octave's implementation using numpy and scipy's fftconvolve.

# Based on Octave implementation by Benjamin Eltzner 

import numpy as np
from scipy.signal import fftconvolve


def normxcorr2(template, image, mode="full"):
    # If this happens, it is probably a mistake    
    if np.ndim(template) > np.ndim(image) or \
            len([i for i in range(np.ndim(template))
                 if template.shape[i] > image.shape[i]]) > 0:
        print("normxcorr2: TEMPLATE larger than IMG. Arguments may be swapped.")

    template = template - np.mean(template)
    image = image - np.mean(image)

    a1 = np.ones(template.shape)
    ar = np.flipud(np.fliplr(template))

    out = fftconvolve(image, ar.conj(), mode=mode)
    image = fftconvolve(np.square(image), a1, mode=mode) - \
            np.square(fftconvolve(image, a1, mode=mode)) / (np.prod(template.shape))

    # Remove small machine precision errors after subtraction
    image[np.where(image < 0)] = 0
    template = np.sum(np.square(template))
    out = out / np.sqrt(image * template)

    out[np.where(np.logical_not(np.isfinite(out)))] = 0
    return out

Using fftconvolve as it is significantly faster than convolve2d or correlate2d. Also allows for N-dimensions.
Python file available on GitHub: normxcorr2-python

[HowTo] Install pfSense 2.1 amd64 on Xen 4.3 as PV HVM

I have been using a HVM pfSense 2.0.1 on Xen 4.1 for the last year or so with no problems and have absolutely loved it, I tried a high end router for 200+ clients and found it was outgunned within 10 minutes, pfSense on the other hand has turned out to be cheaper, VERY stable and with Xen I can use my server for other stuff as well. With the recent releases of Xen 4.3, and pfSense 2.1 with the new Captive Portal upgrades and its move to FreeBSD 8.3 I decided it was time to upgrade stuff.

As requested I have uploaded a working PVHVM image for 64 bit pfSense 2.1. You should be able to use xl console with this when installing.
pfSense 2.1 64-bit PVHVM: pfSense-memstick-2.1-RELEASE-amd64-20131120-1509.img.gz
config file used to create this image: pfSense_SMP.8
if_re.ko based on Realtek driver version 1.83 for FreeBSD (amd64): if_re.ko

If you are reading this blog because xl console doesn't work for you skip to part 5. If you can't get to the installation from VNC especially after it loads a few things then seems to be hanging, try changing the VNC format to raw with less colours. If you problem is the Realtek drivers in FreeBSD go to Extra at the end of the blog.

After almost 3 weeks of tinkering with devwiki.pfsense.org/DevelopersBootStrapAndDevIso I was finally been able to get a stable pfSense 2.1 RC0 build up and running. So far I have found 2 things that don't work. The first one is a big deal, ALTQ is not supported by the xen drivers. The other is being able to use a keyboard when using VNC to the pfSense.

Give your builder DomU plenty of space, RAM and vcpu power to hurry things up. You will need around 5.5GB of space.
Making hvm requires following in builder.cfg

firmware="/usr/local/lib/xen/boot/hvmloader"
builder='hvm'

Install FreeBSD 8.3 (express/minimal you are going to get latest ports anyway)
If you are new to FreeBSD, to get ssh up and running make sure you select minimal and not express.
When it asks you to set up a user account make sure to add them to other groups = wheel
Enabling sshd is the same business (uncomment listen port in /etc/ssh/sshd_config and then /etc/init.d/sshd start)

This is a bit out of date now!
FreeBSD 8.3 is no longer supported. Do not use any ports after rev. r35298*5*
Using git:
git clone https://github.com/freebsd/freebsd-ports.git /usr/ports
git checkout -b 8.3-compatible 14ce842120fdd1ffdce18f634bcf3696ef4995f9
Using svn: 
svn checkout -r 352985 svn+ssh://svn.freebsd.org/ports/head /usr/ports

Please follow the instructions at: https://forum.pfsense.org/index.php?topic=76132.0 to get access to the tools repository.

1. Start out by following the instructions on the devwiki. However change https:/ to git:/ for freesbie2

su
echo "WITHOUT_X11=yo" >> /etc/make.conf
echo "OPTIONS_UNSET=X11" >> /etc/make.conf
echo "BATCH=yo" >> /etc/make.conf
mkdir -p /home/pfsense/pfSenseGITREPO /usr/pfSensesrc
portsnap fetch extract
cd /usr/ports/textproc/expat2 && make depends install
cd /usr/ports/devel/git && make depends install
cd /usr/ports/sysutils/fastest_cvsup/ && make depends install
# tcsh requires reloading available binaries
# if you run this command and it fails -- don't worry.
rehash
cd /home/pfsense && git clone git://github.com/pfsense/pfsense-tools.git tools
cd /home/pfsense && git clone git://github.com/pfsense/freesbie2.git freesbie2
cd /home/pfsense/tools/builder_scripts && chmod a+rx *.sh
csup -h `fastest_cvsup -c tld -q` /usr/share/examples/cvsup/standard-supfile 

Step 2 is no longer necessary with updated tools. However I would recommend the vi part.
2. I would not recommend installing any other editors, I have had build problems every time I installed anything. So stick with vi editor.
vi being one of the most unintuitive editors a cheat sheet helps.

You will also have to edit the BSDinstaller script to look in git instead of https.

vi scripts/get_bsdinstaller.sh
# Find https:/ - You can type /string to find a string in vi
# Change it to git:/ - Press escape and press x to delete character.
# Press i to insert
exit vi
# Press escape and type :x or :x! for read-only files

3. If you get a dfuife_curses error this is the culprit. You MIGHT get it later on when building an iso. Which is quite frustrating since its after almost an hour and a half of build time.
Gives dfuife_curses missing error if not fixed.

vi scripts/rebuild_bsdinstaller.sh
# Add the following line after: mv /etc/make.conf /tmp/
echo "MAKE_JOBS_UNSAFE=yes" >> /etc/make.conf
exit vi

Step 4 is also no longer necessary as the tools are updated to do the same. However I am keeping it in here in case someone wants to make a 2.2 Alpha
4. RELENG_2_1 just got branched. So HEAD now refers to RELENG_2_2 (to make 2.2 ALPHA which is quite unstable)
We have to fix set_version.sh to get the right branch.

cd /home/pfsense/tools/builder_scripts
vi set_version.sh
# find RELENG_8_3)
# In that function:
# Change PFSENSETAG=HEAD to PFSENSETAG=RELENG_2_1
# Can change PFSENSE_VERSION=2.1-BETA to PFSENSE_VERSION=2.1-RC0
# (looks like its only symbolic, doesn't really change any scripts)
exit vi

5. The following enables the serial console on ttyu0, required for xl console.

vi builder_common.sh
# find LOADER_CONF_CUSTOM
# Insert the following line after: install_extra_loader_conf_options() {
sed -i "" -Ee "s/^ttyu0.*$/ttyu0 \"\/usr\/libexec\/getty bootupcli\" cons25 on secure/" ${TTYS}
exit vi
vi loader.conf.local
boot_multicons="YES"
boot_serial="YES"
console="comconsole,vidconsole"
exit vi

6. Add Xen stuff to kernel config

cd conf/kernel
# Add the following lines to pfSense*.8* depends on your TARGETARCH in pfsense-build.conf
# If you are not sure just add that to all files with pfSense and 8 in them 

# virtualization stuff
# adaptive locks probably don't work right with virtualization. Disable them.
options         NO_ADAPTIVE_MUTEXES
options         NO_ADAPTIVE_RWLOCKS # I read somewhere Xen 4.3 has support for this. 
# Not 100% sure though. Also not sure what it even does.
options         NO_ADAPTIVE_SX 

# Xen stuff
options         XENHVM
device          xenpci

7. Now we can get back to the devwiki and finish building iso.

cd ../..
./menu.sh
# Select Set Version -> Select RELENG_8_3. everything else is just default.
exit menu

vi pfsense-build.conf
# Add the following:
export LOADER_CONF_CUSTOM="/usr/home/pfsense/tools/builder_scripts/loader.conf.local"
exit vi

./menu.sh
# Select Apply kernel patches
# Select Rebuild BSDInstaller
# Select Build ISO (selecting developers ISO does nothing as far as I can tell.) 

Once the build iso is completed you will have the iso at /tmp/pfSense/ and the kernel at /tmp/kernels/

Extra. If you are using any Realtek other than a Realtek 8139+ you should probably do the following things.

vi /sys/amd64/conf/GENERIC
# find the following line and press dd. you can probably search for 8111 to make life easier.
device re
# Personally I also removed device rl, just incase.
exit vi

# Go to Realtek Drivers and download the FreeBSD drivers
# I have had success with the v180 drivers on pfSense 2.01. 
# My stock kernel RealTek drivers definitely dont work with 2.1 RC0. 
# and I am going to try the v183 now. Update: working well with pfSense 2.1!

# The readme provided is a bit confusing but I found it easiest to just extract the files
rm -rf /usr/src/sys/dev/re
mkdir /usr/src/sys/dev/re 
cp if_re* /usr/src/sys/dev/re/
rm -rf /usr/src/sys/modules/re
mkdir /usr/src/sys/moduels/re
cp Makefile /usr/src/sys/modules/re/
cd /usr/src/sys/modules/re/
make clean
make
cp if_re.ko /mnt/v180 # To a usb drive or something
# If you only have remote access you can probably also add the files to 
# builder_scripts/conf/copylist/copy.list.RELENG_2_1.
# I would just copy all versions to the usb drive and then try them one after the other.

Proof that it is running under PVHVM

dmesg | grep | "xn"
xn0: at device/vif/0 on xenbusb_front0
xn1: at device/vif/1 on xenbusb_front0
xn0: backend features: feature-sg feature-gso-tcp4
xn1: backend features: feature-sg feature-gso-tcp4
xn0: 2 link states coalesced
xn0: link state changed to UP
xn1: 2 link states coalesced
xn1: link state changed to UP

I can also use xl shutdown pfSense in Dom0 which doesn't work on HVM.

Speed wise iperf tests have not seen a massive improvement over HVM. Throughput of 540 Mbps on HVM vs 590 Mbps on PVHVM. Although I would have to check CPU loads as well to see how much better it actually is.

ALTQ for xen drivers This is just a plea to anyone with the knowledge of drivers under FreeBSD to take a look at implementing it.
If anyone is interested in adding ALTQ support, take a look at the man page for it, seems to be pretty helpful on what needs to be changed.  Also from what I could understand of it the driver that would need to be changed is netfront.c in src under /sys/dev/xen/netfront/netfront.c

Let me know how it works out for you guys, if there is other stuff not working and if I could improve anything. Thanks for reading!

My thread on pfSense forums: click here