More Git Tips - Pair Programming Setup

Nathan and I forked Nick's appletoo repo and can push/pull changes from our own branches.


=========================
A Few Useful Commands
=========================
Push upstream
git push -u origin [NAMEOFBRANCH]

Pull from specific remote server
git pull [REMOTESERVERNAME] [BRANCHNAME]

Add remote server
git remote add [LOCALSERVERNICKNAME] [URL]


=========================
A Few 'Oh-Duh' Commands
=========================

Check possible remote servers
git remote


Clear staging area
git reset

Check status
git status

Check differences
git diff

Exploitation -- The way I like it.

========================================================
Gibberbot
Possible Project / Contribution Candidate
========================================================
Notes
+  Uses XTPP protocol for messaging 
+  Follow Gibberbot GitHub instructions
+  Helpful guide to fork-a-repo

FollowUp: Steffi continued on this project. Mostly code clean up. Project - Interest mismatch.


========================================================
Android
Area of Interest / Possible Project

========================================================

Notes
+  Mission = root an Android device using any available means.
+  Sub-Mission: Define 'available means.'
+  Downloaded Android Source (here)
          o  Takes at least an hour to grab everything. 

Operating Systems Lab 1:

"One interesting case I found in my research was a program that set its current directory to /etc/cron.d (where scheduled tasked are stored). The hacker requested a core dump happen in the case of an emergency crash of his process, then recruited another process to kill it. The dump occurred, and chron treated it as a text file. When it was next run, because chron had root privileges and was running the process, the process also had root privileges."

... Inspiration? ....




Understand a Pre-Existing Exploit
Eric and I traced this code, which successfully rooted Android v2.2
ARM Stack looks like this

FPGA + Verilog

Board Details :

PICO Computing E-101 

• (E101_Quick_Ref.pdf) 

• Spartan 6 data sheets here 

• "Data Sheet" here (crap) 

• Chip Data Sheet here 

• std_logic values 

Good for cryptography/cryptanalysis, huh? ... huuuuuh..... Zodiac killer project application... ?!? :D

================//==================

Using "VHDL: Learn By Example" to Understand VHDL:

Entity ENT_NAME is
port(    VAR1: in std_logic;
            VAR2: out std_logic_vector(7 downto 0);
             ...
             VARN: in std_logic               <--------------NO ";" for last entry
);
end ENT_NAME;
architecture ARCH_NAME of ENT_NAME is
    component OTH_ENT_NAME is
    port(    A:   in std_logic;
                ...
                B:   in std_logic                     <-------------------NO ;
     );
     end component;
    signal SIG_NAME: std_logic;
    ...
begin
    process( INP_VARS....)
    begin
    [LOGIC GOES HERE];
    end process;
    GATE_NAME: ENT_NAME port map (COMP_VARS=>ENT_VARS);
end ARCH_NAME; 

VHDL QUESTION:     Q: When is a process necessary? For multi-line logic only? For dependencies or certain operations?
     A: Processes are required to reuse code. (Change in parameter values triggers re-run).

 ================//==================

Verilog

Using "Verilog: Learn By Example"

Blocking vs. Non-Blocking assignments  

+ = Blocks procedural flow (Assignment happens first. Rest blocked)        

+ Use for time sensitive     

+ <= Doesn't         + Can be executed w/o dependency/timing issues        + Several register assignments in same time step        + Resembles actual hardware more Verilog QUESTIONS:     Q: When is it necessary to list "@ always" parameters with "or" vs. a comma?

BLOGGER IS MESSING UP MY FREAKING FORMATTING. ^ GAAAAH
 ================//==================

Driver Issues:

Install Digilent Adept System (at least v2.4)Install Plugin (follow pdf directions)answers here maybe? http://www.xilinx.com/support/answers/30184.htmDocumentation we were following for  DRIVERS http://www.xilinx.com/support/documentation/user_guides/ug344.pdf

NOPE^

I HATE DRIVERS.


But not really.


Please work.


I'm soorrrrryjustworkplzstopthetortuuuureee....


*********USE THIS DRIVER - PICK 64 BIT EVEN IF ON 32 BIT ****************


 ================//==================


Immediate Goal: Run and understand this example.
- Helpful when adding pins...kinda...notreally....
- Testbench related here




 ================//==================


Misc Resources:

TestBench TutorialVerilog 

TutorialVerilog Workflow (Work flow, colorful tutorial)

Misc:

WANT (to make) --> Noisy Jelly 

Understanding RSA

Museum Pleasantries and Girlish Babble - Disregard

Brooklyn Museum on Saturday with some 622er's - Nathan, Lisa, David C., Vicki, and James (who recognized and finished a Gershwin song quote I started-- Awesome Points awarded.)

Note to self - read more. Titles mentioned = The Gods Themselves, The Last Question, Snow Crash, Dune (which I have at least heard of!)

Apparently, alcoholic brunches are "a thing," here. Between certain hours, certain restaurants serve bottomless brunch drinks. I had to get the full "New York experience," so I tried it...

Brunch:

• One (bottomless) mimosa
• Screwdriver 
• Dagwood-style hamburger concoction 

After brunch, I went shopping for clothing with coding buddies  (whendoesthiseverhappen?!) - Martha, Jane, Ingrid, and Trucy. We found a cool dark corner at  a bar, drank (water)... 

Girl talk and interview advice - Never name your price. They can give you more, so ask. Haggle. Assert yourself. Show that you know your worth - and you are worth more than they are offering! Strong independent women. Yea, I think I'm in the right place ;)

Hacker School Game night was funnn..... I left early. Tired and semi-drunk all day. 


The whole 'intoxicated thing' is not as great as it's cracked up to be. I'd rather act drunk while sober (in some silly dancing Broadway theater manner) than act drunk while actually drunk... or unintentionally act drunk while sober. Najva lent me the first episode of Season III of the Real L Word the other day. I don't follow it, but that's what I thought of all the characters. Either they are drunk on set or they are just permanently intoxicated. Or just really stupid.


OR they're utterly brilliant, witty, and intelligent in that good-ol American way, and I just don't see it. Right, I'm sure that's the case.

...


Ok, fine, maybe I'm just bitter because there are no nerds. 

Nyquist-Shannon Sampling + Shazam

Nyquist–Shannon sampling theorem
    MyNutShellDefinition: With twice the frequency of the highest contributing sine wave frequency, you can accurately reproduce your signal. 


- Following the Shazam Algorithm - 
The Concept:

If you put the constellation map of a database song on a strip chart, and the constellation map of a short matching audio sample of a few seconds length on a transparent piece of plastic, then slide the latter over the former, at some point a significant number of points will coincide when the proper time offset is located and the two constellation maps are aligned in register. 

The Implementation:

Each anchor point is sequentially paired with points within its target zone, each pair yielding two frequency components plus the time difference between the points (Figure 1C and 1D).  These hashes are quite reproducible, even in the presence of noise and voice codec compression.  Furthermore, each hash can be packed into a 32-bit unsigned integer.  Each hash is also associated with the time offset from the beginning of the respective file to its anchor point, though the absolute time is not a part of the hash itself.

Questions:

What constitutes a constellation point? (There may/will be multiple per bin. We will use one.)

MISC: 

DFT Walkthrough - (think I saw his stuff in Modeling&Post)

Using the sampling rates etc suggested here


In some audio processing approaches, a visual representation of the signal is created and analyzed. (Note to self - check out Hough Transform )

After presentations, jLaster mentioned Bloom Filters, which rapidly tell you whether an element is present in your set. Might be useful for a preliminary test on very large data sets. 

RANDOM : 
What was that, Eleanor dear? - marthakelly's hilarity
Beautiful Sudoku Code - recommended by jczetta

Python Requests - Logging and Expect Headers

Follow-up on Requests Issue #695 (the Logging Bug):

Muhtasib noticed Requests doesn't implement logging.
Taking a closer look at urllib3 code & documentation:
        intro to logging        Tutorial (more detailed)
   Oauth also uses logging...
    

Action Taken - Comment to K.R., asking if/how this should be implemented.

Starting on Requests Issue #713 (Support for Expects Http Header Bug)

Requests can check the request headers, and if it finds an "Expect" header it will wait for the 100 Continue response from the server. If it does not come, this error should flow to the caller in a way that it can distinguish a network problem from a failed expectation error.

You send an Expect header, telling the server you expect a 100 (continue) response before you will send the body. If there are images or videos or some form of substantial data in the body, you don't want to send if the server will reject it anyway.

Because you are waiting for a response from the server, one request actually becomes two - a request sending the header and a request sending the body.

Dialogue goes a little something like....

Client: I want to post something. Ready? 

Server: Sure! [100-continue]

Client: Here it comes! *posts*

Server: Got it! [200]

Plan of Attack


if request.headers contains Expect-header 
    send request with header only
    wait for response


    if response == 100 (continue)
        send request with body
else
    send request as normal

Things to consider:

• limit wait time and throw exception/error if no response
• if server prematurely closes ... don't try to send again

Testing - We want to find a website that give a 100-continue. We want to see that 100-continue. HOW? In the response header? In the response content? 

Wandering

"So, I was wandering down Broadway the other day, and ... "


     (Oh geez, I have been waiting to use that line)

"...and I happened upon a playbill for Gore Vidal's The Best Man...."

"...starring James Earl Jones and Angela Lansbury. My little theater-heart, aflutter with curiosity, dragged me inside. I emerged with a single glorious ticket to a single glorious matinee. 

Even Charlie would have been jealous."

"When I returned an hour later, the theater was gorged with patrons. I imagined my seat - two hours, condemned to the back wall... "

"Nope.

8D

A private box! So close I could see the human details of the talent -- Dimples on smiles and whites of the eyes, crevices of hands and wisps of hair. I could have reached down and brushed Legends. But there was no need. 

They reached up to me, instead."

dawwwww :)

</cheesy>

Lincoln Center hosts "MidSummerSwing" dance events for three weeks. I caught the last hour of the last night of tango. Outside, with live music, stage erected from scratch and a dance floor that costs $20 to step on. I stayed on the concrete... The bottoms of my shoes hate me for this >.< 

I danced with a single partner named Eduardo, who supposedly teaches and worked for Fred Astaire earlier in his life. Flashy, spinning all over the dance floor, and he taught along the way. He gave me his card, eager for another student. Not sure if I'll follow up... but it was some great dancing. Dips and ganchos and slides and extensions and such quick footwork!

You. Will. Be. Ex.TeRRminated. (BugHunting)

Open Source Week Project = Requests: HTTP for Humans 

Requests Issue #567 (SSL Documentation Bug)
Document use cases of Requests 'get' function with and without SSL certification verification.

What is SSL?
Secure Socket Layer.
https://
.pem
.crt + .key
client-side certification no longer used.
    ( confirmed by speaker via gsterns HOPE speaker-searchchrome )
Submitted patch


Resources
    Diffie Hellman Key Exchange
    SSL - all you wanna know (search for "Well, we really need to talk about TLS.")

Testing - tgebru configured her Apache server + SSL certification

Requests Issue #699 (Differentiate between Timeout Errors)
Connection Pool Error or Socket Timeout Error? We fixed this by "bubbling" the error message sent by urllib3 through Requests to the user.
Submitted patch

Requests Issue #695 (Document Logging)

GIT Random Scratchy Notes...

make changes on separate branch from master
(already forked and made changes)
check what branch
git branch
(sugg = develop branch, if one)
Fork
git checkout master
THEN
git checkout -b name-of-your-branch
(which will branch from master, since you did that before)
git commit (message relevant to checkin only. details in pull request)
git push origin Branch-Name_you_created
submit pull request on github, include details here.
git diff - see changes
Git Help http://gitref.org/creating/

RANDOM
http://www.fivefeetoffury.com/2012/06/25/get-pregnant-by-25-if-you-want-a-high-powered-career/

Sunbathing

I've been out in the sun a lot. 

Hope I don't get tan lines. 

XD

</dork>

Crown Heights IV

I moved....

Goodbye, favorite Sketchy-Bakery-Across-the-Street.

The new place.

The lobby.

My room (much larger than it looks).

My room from the other side.

It's very long, though you can't tell in the photos. Enough room to dance! 

The view.

Vegetarian House Dinner, prepared by Simona.

No lactose :)

Shar and I went to NJ to watch the fireworks from Hoboken. Spectacular, with some reflective moments. Beautiful day, great company, serenity in enjoying the sea of people. Another open-minded one. 

This doesn't do it justice, but imagine....

+ Frozen yogurt afterwards.

Like Dandelion Wine. 

FFT Pix

Plotting Input WAV file

from pylab import *

import matplotlib.pyplot as plt

from scipy.fftpack import fft

import wave

import numpy as np

file = "toy-rattle-2.wav"

spf = wave.open(file,'r')

f = spf.getframerate()

sound_info = spf.readframes(-1)

sound_info = fromstring(sound_info, 'Int16')

plt.plot( sound_info )

plt.show() 

 Plotting the F.T. of sound_info (complex numbers)

ft = fft( sound_info )

Magnitude of F.T.

ft_mag = map( abs, ft )

Applying a Windowing Function ( Hanning )

window = np.hanning( len(sound_info ) ) 

Apply the windowing function before taking the FFT: (pythonically)

sound_window = []
for s, w in zip( sound_info, window ):
   sound_window.append( s * w ) 

ft = fft( sound_window  )
ft_mag = map( abs, ft )

It does not seem like much difference, other than slight variations noticeable on the edges. Scale might be affecting it.

ft_u = fft( sound_info )ft_w = fft( sound_window )ft_um = map( abs, ft_u )  ft_wm = map( abs, ft_w ) 

dif = []
for w, u in zip( ft_wm, ft_um):
    dif.append( abs( w -  u ) )

Below, I graphed the difference:

Which doesn't look like any difference. This baffles me...

Specgram Shazam - Research

AudioLab Documentation

Power Spectral Density

Intro to PSD ( Human Friendly ) 

INSPECTING THE SPECGRAM FUNCTION
Returns the following:

Pxx - "a len(times) x len(freqs) array of power" aka Periodogram. Pxx is the estimation of the DFT of a function.

WOAH ASSUMED WRONG RETURN ORDER - documentation lists two different orders. We were looking at the wrong one. Now the data association makes sense - each item in the Pxx array corresponds to a time (where in the input it originated from), and there is a single array which describes the frequencies from the (overall) input's FFT.


Eesh!


FINDING A HASHING FUNCTION  
Shazam hashes short segments - 30 peaks per second are hashed and saved. Target zone hashes are generated from the input and queried against the database. We discussed hashing the entire song based on peak distance. Peak 1 to peak 2, peak 2 to peak 3... but  "when your target hash value has fewer bits than what's being hashed, then uniqueness cannot ever be guaranteed." Length is an issue - we would either have to enforce a given length or squeeze enough data out of a very short song to match the data from the longer song. This could lead to false positives. I don't like it. 


How do we create a function that is, as the Shazam paper suggested is necessary, reproducible independent of position within an audio file?


===========================================

FAST FORWARD - JUL 6
===========================================

Decision made to implement the Shazam algorithm.

Tango in Union Square

I merrily made my way to the Big Apple Tango Fest Farewell Milonga... and found this.

WTF. I've had nothing but trouble finding tango around here.
I walked to Union Square...



....saw cool clocks along the way....



...and thankfully found a milonga. Recorded music, unfortunately, but who's judging ;) </spoiled-tanguera-missing-Q-Tango>

I snuck in and took a seat. A gentleman named Peter, also new to Sunday milongas, asked me to dance first. He was concave. It felt like he was running away from me at times, but enjoyable. 

Then Charlie, flashy, quick, hands shifting as though he were 'steering' or.... something... probably looked impressive. Passionate and energetic! But so much energy to follow. 


Then, Alex the Ukrainian. Wonderful embrace! The safe warm kind that melts into a hug at the end. (Bill....? Biiilllll.... ) For the second song, he became a stronger, more powerful lead. Totally different feel. The third dance was a learning experience. He worked with me on some on volcadas - I have trust issues with those. Hopefully, I'll run into him again on the dance floor. He suggested the outdoor milongas - Sunday and Wednesday. Really great!

Not a huge crowd. According to Alex, only four live tango bands in NYC. He's been on the scene since the 80s. I'd expect a bigger scene, but who knows. Maybe the festival people were off having their own party. As it were, Svet would have been queen there ;) I felt kind of princess-like, myself. I don't know if this is the normal crowd, though. I only got three tandas in before it ended at 10pm. 


Then, I wandered. 


Bought spoon ring from a local artisan. Old PanAm spoon. Yep.... 

Union Square is dreamy that time of night after a milonga, coconut juice in hand. 

DSP - Investigations

RESEARCH

DEFINITIONS - LITE:

A digital signal processor (DSP) is a specialized microprocessor with an architecture optimized for the fast operational needs of digital signal processing. --wikipedia

Digital signal processing (DSP) is the mathematical manipulation of an information signal to modify or improve it in some way. -- wikipedia

In signal processing, a filter is a device or process that removes from a signal some unwanted component or feature. --wikipedia

Fourier Transform => Say you represent an input function in terms of sine waves. The FT is like an accountant, showing how much each sine wave at frequency (whatever) contributes to the overall reproduction. --me, after looking at this ... finally... 

        Our DSP goal would be filtering. Isolate individual elements and analyze them. How do we detect those elements? Familiar filters - Low pass, Hi pass, Band filter. These determine which signals are allowed to pass though from input to output. Can elements of our input be accurately differentiated based on frequency alone? Not yet convinced...


Our input would be analog from a mic, I assume. Do/should we convert to digital? 

RESOURCES 


MIT DSP Course

• Assumes knowledge of 
• linear system theory for continuous-time  signals  and  systems
• Fourier Transforms
• Laplace Transforms

Implementation Tools (SO)

Fundamentals of Signal Analysis - GREAT resource, but I started getting too sucked in. This is a shift from the academic world. Implementation does not require 100% from scratch understanding... I always feel the drive to go that route. That's what partners are for, though - kick start reality reminders!

Rhythm Detection in Recorded Music

Automated Measurement of Song Similarity

IDLE BRAINSTORMING:

Generally, DSP's are dedicated integrated circuits; however DSP functionality can also be produced by using field-programmable gate array chips (FPGA’s). -- wikipedia^

Hmm.... VHDL + Signal processing.... :D

We could implement a signal comparison library and implement something like Shazam (explained here, official paper here).

From my understanding, the algorithm is as follows: 

• Find the peaks
• Pair the peaks with a designated anchor point
• Generate a hash using characteristics of the pair, including distance between peak and anchor
• Query database for a series matching keys that occur with temporal locality (aka several keys appear close together)  

This algorithm is currently protected under... copyright, I suppose. Python implementations of it are not free to be published, but coders online claim to have implemented this in under two hours with 100 lines of code. 

In other words, tomorrow we ARE finishing this.

PROGRAMMING

Goal: Run THIS DEMO and understand it.

• virtualenv sig_env
• cd sig_env
• pip install scikits.audiolab
• pip install numpy
• brew install gfortran (for scipy because I'm on lion)
• pip install scipy
• pip install matplotlib

(outside of sig_env)

• sudo pip install matplotlib
• sudo pip install scipy
• (sudo pip install numpy? already had it)

The following line of code : 

spectrogram = specgram(sound_info, Fs = f, scale_by_freq=True,sides='default')

is responsible for generating the spectrogram.

Here is my annotated version of it: 

AS EXPLAINED TO D.PETER

smarg

anyhow, you define some delta t to look at on this graph.

(I'm actually describing the input graph. The spectrogram comes second)

[camera zooms in on only that section of the squiggly wave]

You say, "hmmmm, it sort of looks like a sin wave"

or "hmmm, it looks like sums of sine waves:

So you then take the Fourier Transform of that tiny little piece...

aka you express it in sine waves and....

hmmm how do I explain this....

It's as though the Fourier Transform is an accountant, who tallies how much each sine wave contributed to the overall result.

So if a sine wave at frequency X contributed more than a sine wave at frequency Y, X has a higher value in the Fourier Transform

Making sense?

dpeter

ha pretty nice analogy

smarg

^_^

dpeter

i still dont completely get it

but still

xD

smarg

Well for each (delta t) step you take across the original graph, you get a whole new graph telling you the most influential sine waves used to recreate that part of the original function.

dpeter

i see

smarg

it would be nice to have a 3d graph of it... the spectrogram that we were looking at showed the power for each (delta-t, frequency) pair

dpeter

the fourier transform tells you the individual sine waves?

or what

smarg

so, you know the derivative tells you about how the original function changed at that point?

The Fourier Transform tells you about how the tacky-rip-off-fake-SINE-WAVE version of the function was created.

It's not point to point specific, but... yea.

It tells you about re-creation, not change.

Maybe that wasn't a good comparison

I'm going to try making a picture.

Which you see above!

Gosh I love just copying and pasting things. And people who will just chat about awesome sciency things on their spare time! YAY!!

This was followed by a discussion about the Julia Set. Trippy and awesome... and I totally knew what it was when it came up. Thanks, reading group!