nf_conntrack: table full, dropping packet on Nessus server

Issue caused by having iptables rule/s that track connection state. If the number of connections being tracked exceeds the default nf_conntrack table size [65536] then any additional connections will be dropped. Most likely to occur on machines used for NAT and scanning/discovery tools (such as Nessus and Nmap).

Symptoms: Once the connection table is full any additional connection attempts will be blackholed.

 

This issue can be detected using:

$dmesg
nf_conntrack: table full, dropping packet.
nf_conntrack: table full, dropping packet.
nf_conntrack: table full, dropping packet.
nf_conntrack: table full, dropping packet.
...

Current conntrack settings can be displayed using:

$sysctl -a | grep conntrack
net.netfilter.nf_conntrack_generic_timeout = 600
net.netfilter.nf_conntrack_tcp_timeout_syn_sent = 120
net.netfilter.nf_conntrack_tcp_timeout_syn_recv = 60
net.netfilter.nf_conntrack_tcp_timeout_established = 432000
net.netfilter.nf_conntrack_tcp_timeout_fin_wait = 120
net.netfilter.nf_conntrack_tcp_timeout_close_wait = 60
net.netfilter.nf_conntrack_tcp_timeout_last_ack = 30
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 120
net.netfilter.nf_conntrack_tcp_timeout_close = 10
net.netfilter.nf_conntrack_tcp_timeout_max_retrans = 300
net.netfilter.nf_conntrack_tcp_timeout_unacknowledged = 300
net.netfilter.nf_conntrack_tcp_loose = 1
net.netfilter.nf_conntrack_tcp_be_liberal = 0
net.netfilter.nf_conntrack_tcp_max_retrans = 3
net.netfilter.nf_conntrack_udp_timeout = 30
net.netfilter.nf_conntrack_udp_timeout_stream = 180
net.netfilter.nf_conntrack_icmpv6_timeout = 30
net.netfilter.nf_conntrack_icmp_timeout = 30
net.netfilter.nf_conntrack_acct = 0
net.netfilter.nf_conntrack_events = 1
net.netfilter.nf_conntrack_events_retry_timeout = 15
net.netfilter.nf_conntrack_max = 65536
net.netfilter.nf_conntrack_count = 1
net.netfilter.nf_conntrack_buckets = 16384
net.netfilter.nf_conntrack_checksum = 1
net.netfilter.nf_conntrack_log_invalid = 0
net.netfilter.nf_conntrack_expect_max = 256
net.ipv6.nf_conntrack_frag6_timeout = 60
net.ipv6.nf_conntrack_frag6_low_thresh = 196608
net.ipv6.nf_conntrack_frag6_high_thresh = 262144
net.nf_conntrack_max = 65536

To check the current number of connections being tracked by conntrack:

/sbin/sysctl net.netfilter.nf_conntrack_count

Options for fixing the issue are:

  1. Stop using stateful connection rules in iptables (probably not an option in most cases)
  2. Increase the size of the connection tracking table (also requires increasing the conntrack hash table)
  3. Decreasing timeout values, reducing how long connection attempts are stored (this is particularly relevant for Nessus scanning machines that can be configured to attempt many simultaneous port scans across an IP range)

 

Making the changes in a persistent fashion RHEL 6 examples:

# 2: Increase number of connections
echo "net.netfilter.nf_conntrack_max = 786432" >> /etc/sysctl.conf
echo "net.netfilter.nf_conntrack_buckets = 196608" >> /etc/sysctl.conf
# Increase number of bucket to change ration from 1:8 to 1:4 (more # memory use but better performance)
echo 'echo "196608" > /sys/module/nf_conntrack/parameters/hashsize' >> /etc/rc.local

# 3: Alter timeout values
# Generic timeout from 10 mins to 1 min
echo "net.netfilter.nf_conntrack_generic_timeout = 60" > /etc/sysctl.conf

# Change unacknowledged timeout to 30 seconds (from 10 mins)
echo "net.netfilter.nf_conntrack_tcp_timeout_unacknowledged = 30" > /etc/sysctl.conf

# Change established connection timeout to 1 hour (from 10 days)
echo "net.netfilter.nf_conntrack_tcp_timeout_established = 3600" > /etc/sysctl.conf

These changes will persist on reboot.

To apply changes without reboot run the following:

sysctl -p
echo "196608" > /sys/module/nf_conntrack/parameters/hashsize

To review changes:

sysctl -a | grep conntrack

Reference and further reading: http://antmeetspenguin.blogspot.com.au/2011/01/high-performance-linux-router.html

Setting secure, httpOnly and cache control headers using ModSecurity

Many older web applications do not apply headers/tags that are now considered standard information security practices. For example:

  • Pragma: no-cache
  • Cache-Control: no-cache
  • httpOnly and secure flags

Adding these controls can be achieved using ModSecurity without any needs to modify the application code.

In the case where I needed to modify the cookie headers to include these now controls I added the following to core rule set file: modsecurity_crs_16_session_hijacking.conf.

#
# This rule will identify the outbound Set-Cookie SessionID data and capture it in a setsid
#
#addding httpOnly
Header edit Set-Cookie "(?i)^(JSESSIONID=(?:(?!httponly).)+)$" "$1; httpOnly"
Header set Cache-Control "no-cache, no-store, must-revalidate"
Header set Pragma "no-cache"
Header set Expires "0"

 

This adds the cookie controls we were after – Depending on your web application you may need to change ‘JSESSIONID’ to the name of the relevant cookie.

You can find the cookie name simply using browser tools such as Chrome’s Developer Tools (hit F12 in chrome). Load the page you want to check cookies for, click on the Resources tab:

ChromeCookies

After setting the HTTPOnly and Secure flags you can check the effectiveness using the Console table and listing the document cookies… which should now return nothing.

document.cookie

Migrating to EJBCA from OpenSSL and TinyCA

Install and configure EJBCA

EJBCA 6.0.3 – http://www.ejbca.org/download.html

JBoss AS 7.1.1 Final – http://download.jboss.org/jbossas/7.1/jboss-as-7.1.1.Final/jboss-as-7.1.1.Final.zip

Prereqs:

Ref:

Detailed deployment guide: http://majic.rs/book/free-software-x509-cookbook/setting-up-ejbca-as-certification-authority

EJBCA doc: http://ejbca.org/adminguide.html

Architecture

Recommended architecture (source: http://ejbca.org/architecture.html)

Import existing OpenSSL CA

Step 1 – Export the OpenSSL priv key and cert to a PKCS#12 keystore:

openssl pkcs12 -export -out exitingCA1.p12 -inkey <existingCA1.key.pem> 
        -in <existingCA1.crt.pem> -name existingCA1

Step 2 – Import the PKCS#12 keystore to EJBCA CA

<ejbca_home>/bin/ejbca.sh ca importca <caname> existingCA1.p12

Step 3 – Verify import

<ejbca_home>/bin/ejbca.sh ra adduser

### IMPORTANT ###

Distinguished name order of openssl may be opposite of ejbca default configuration – http://www.csita.unige.it/software/free/ejbca/ … If so, this ordering must changed in ejbca configuration prior to deploying (can’t be set on a per CA basis)

Have not been able to replicate this issue in testing.

Import existing TinyCA CA

Basic Admin and User operations

Create and end entity profile for server/client entities

Step 1 – Create a Certificate Profile (http://ejbca.org/userguide.html#Create a Certificate Profile for SSL servers)

Step 2 – Create and End Entity Profile (http://ejbca.org/userguide.html#Create an End Entity Profile for SSL servers)

* EndEntities can be deleted using:

<ejbca_home>/bin/ejbca.sh ra delendentity <username>

Issuing certificates from CSRs

End entities need to be created for clients/servers that require certificates signed by our CA.

Step 1 – Create and End Entity (http://ejbca.org/userguide.html#Issue a new server certificate from a CSR)

Step 2 – Sign CSR using the End Entity which is associated with a CA

Importing existing certificates

EJBCA can create endentities and import their existing certificate one-by-one or in bulk (http://www.ejbca.org/docs/adminguide.html#Importing Certificates). Bulk inserts import all certificates under a single user which may not be desirable. Below is a script to import all certs in a directory one by one under a new endentity which will take the name of the certificate CN.

#!/bin/sh

# for each certificate in the directory
#       create and enduserentity
#       enduserentity username = certificate CN
#       enduserentity token/pwrd = certificate CN

EJBCA_HOME="/usr/share/ejbca"
IMPORT_DIR=$1
CA=$2
ENDENTITYPROFILE=$3
SSLCERTPROFILE=$4
AP="_OTE"

if [ $# -lt 4 ]; then
        echo "usage: import_existing_certs.sh <IMPORT_DIR> <CA_name> <END_ENTITY_PROFILE> <SSL_CERT_PROFILE>"
        exit 1
fi
for X in $IMPORT_DIR*.pem
do
        echo "######################################################"
        echo "Importing: " $X
        CN=$(openssl x509 -in $X -noout -text | grep Subject: | sed -n 's/^.*CN=(.*),*/1/p')
        echo "CN: " $CN
        printf "Running import: %s ca importcert '%s' '%s' '%s' ACTIVE NULL '%s' '%s' '%s'n" "$EJBCA_HOME/bin/ejbca.sh" "$CN" "$CN" "$CA" "$X" "$ENDENTITYPROFILE" "$SSLCERTPROFILE"
        $EJBCA_HOME/bin/ejbca.sh ca importcert "$CN$AP" "$CN$AP" "$CA" ACTIVE null $X $ENDENTITYPROFILE $SSLCERTPROFILE
        echo "######################################################"
done

Creating administrators

Create administrators that can sign CSR and revoke certificates: http://ejbca.org/userguide.html#Administrator%20roles

Revoking certificates

#Generate CRL via command line
# List CAs
/usr/share/ejbca/bin/ejbca.sh CA listcas
# Create new CRLs:
/usr/share/ejbca/bin/ejbca.sh CA createcrl "<CA name>" -pem <output_file>
# Export CRL to file
/usr/share/ejbca/bin/ejbca.sh CA getcrl "<CA name>" -pem <output_file>.pem

Checking certificate validity/revoke status via OSCP

openssl ocsp -issuer gtld_CA_cert.pem -CAfile gtld_CA_cert.pem 
-cert gtld_registrar5.pem -req_text -url http://localhost:8080/ejbca/publicweb/status/ocsp

Monitoring expiring certs

<ejbca_home>/bin/ejbca.sh listexpired 100

 

Getting started with ModSecurity

XSS, CSRF and similar types of web application attacks have overtaken SQL injections as the most commonly seen attacks on the internet (https://info.cenzic.com/2013-Application-Security-Trends-Report.html). A very large number of web application were written and deployed prior to the trend up in likelihood and awareness of XSS attacks. Thus, it is extremely important to have an effective method of testing for XSS vulnerabilities and mitigating them.

Changes to production code bases can be slow, costly and can miss unreported vulnerabilities quite easily. The use of application firewalls such as ModSecurity (https://github.com/SpiderLabs/ModSecurity/) become an increasingly attractive solution when faced with a decision on how to mitigate current and future XSS vulnerabilities.

Mod Security can be embedded with Apache, NGINX and IIS which is relativity straight forward. In cases where alternative web severs are being used ModSecurity can still be a viable option by creating a reverse proxy (using Apache of NGINX).

How can ModSecurity be used?

  • Alerting
  • Transforming
  • Blocking
  • and more

These functions can be enacted by rules.

A default action can be created for a group of rules using the configuration directive “SecDefaultAction

Using the following SecDefaultAction at the top of rule set that we want enable blocking and transforming on is a blunt method of protection. Redirection can also be used as a method of blocking.

A powerful web application firewall - free software!
A powerful web application firewall – free software!

Example of a default action to be applied by ruleset (note defaults cascade through the ruleset files):

SecDefaultAction “phase:2,log,auditlog,deny,status:403,tag:’Unspecified usage'”

Rulesets have been created by OWASP.

Using optional rulesets,  modsecurity_crs_16_session_hijacking.conf and modsecurity_crs_43_csrf_protection.conf ModSecurity can provide protection against Cross Site Request Forgeries [CSRF]. The @rsub operators can inject a token on every form (and/or other html elements). ModSecurity can store the expected token value as a variable which is compared to the value posted via forms or other html elements. ModSecurity rules can be based on request methods and URIs etc – alongside the ability to chain rules there are a huge number of options for mitigating XSS and CSRF without impacting normal applicatioin usage.

@rsub

Requirements:

  • SecRuleEngine On
  • SecRequestBodyAccess On
  • SecResponseBodyAccess On

## To enable @rsub

  • SecStreamOutBodyInspection On
  • SecStreamInBodyInspection On
  • SecContentInjection On

Injecting unique request id from mod_unique_id into forms:

SecRule STREAM_OUTPUT_BODY "@rsub s/<\/form>/<input type=\"hidden\" name=\"rv_token\" value=\"%{unique_id}\"><\/form>/" \
"phase:4,t:none,nolog,pass"

Some simple rules:

&lt;LocationMatch ".*/&lt;directory requiring authentication&gt;/.*"&gt;
# All requests submitted using POST require a token - not the validation of the token can only be completed if that variable is stored from a previous response
SecRule REQUEST_METHOD "^(?:POST)$" "chain,phase:2,id:'1234',t:none,block,msg:'CSRF Attack Detected - Missing CSRF Token when using POST method - ',redirect:/" SecRule &amp;ARGS:token "!@eq 1" "setvar:'tx.msg=%{rule.msg}',setvar:tx.anomaly_score=+%{tx.critical_anomaly_score},setvar:tx.%{rule.id}-WEB_ATTACK/CSRF-%{matched_var_name}=%{matched_var}"
# Check referrer is valid for an authenticated area of the application</p>
SecRule REQUEST_HEADERS:Referer "!@contains &lt;my website&gt;" "block,phase:2,id:'2345',t:none,block,msg:'CSRF Attack Detected - No external referers allowed to internal portal pages',redirect:/"
SecRule REQUEST_URI "@contains confirmUpdate" "chain,phase:2,id:'3456',t:none,block,msg:'CSRF Attack Detected - Missing CSRF Token. Confirmation button - ',redirect:/" SecRule &amp;ARGS:rv_token "!@eq 1" "setvar:'tx.msg=%{rule.msg}',setvar:tx.anomaly_score=+%{tx.critical_anomaly_score},setvar:tx.%{rule.id}-WEB_ATTACK/CSRF-%{matched_var_name}=%{matched_var}"
&lt;/LocationMatch&gt;

Pros:

  • Wide capabilities for logging, alerts, blocking, redirecting, transforming
  • Parses everything coming into your web server over HTTP
  • Virtual patching – if a vulnerability is made public that affects your web application you can write and deploy a rule to mitigate the vulnerability much faster than re-release of application code patched
  • Extended uses – the capabilities of ModSecurity can be applied to applications outside the scope of application security

Cons:

  • Added complexity to your application delivery chain – another point for maintenance and failure
  • Performance costs? – Though I have not had the opportunity to test the performance costs holding session information in memory and inspecting every byte of HTTP traffic can’t be free from performance cost
  • Hardware costs – Particularly if using ModSecurity’s BodyAccess and BodyInspection features, memory usage will be significant

Improving deployments:

  • Starting off being aggressive on warnings and very light on action is a necessity to ensure no impact on normal application usage
  • From this point rules and actions need to be refined
  • Understanding how the applications works allows the use of ModSecuirtys header and body inspection in effective ways

Some other notes extracted from the ModSecurity Handbook – If you decide to use ModSecurity I strongly recommend buying the handbook. It is not expensive and saves a lot of time.

### RULE STRUCTURE ###
SecRule VARIABLES OPERATOR [TRANSFORMATION_FUNCTIONS, ACTIONS]

### VARIABLES ###
REQUEST_URI Request URI, convert to exclude hostname
REQUEST_METHOD Request method
ARGS Request parameters (read-only collection)
ARGS_NAMES Request parameters’ names (collection)
ARGS_GET Query string parameters (read-only collection)
ARGS_GET_NAMES Query string parameters’ names (read-only collection)
ARGS_POST Request body parameters (read-only collection)
ARGS_POST_NAMES Request body parameters’ names (read-only collection)
### STRING MATCHING OPERATORS ###
@beginsWith Input begins with parameter
@contains Input contains parameter
@endsWith Input ends with parameter
@rsub Manipulation of request and response bodies
@rx Regular pattern match in input
@pm Parallel pattern matching
@pmFromFile (also @pmf as of 2.6) Parallel patterns matching, with patterns read from a file
@streq Input equal to parameter
@within Parameter contains input
### NUMBER MATCHING OPERATORS ###
@eq Equal
@ge Greater or equal
@gt Greater than
@le Less or equal
@lt Less than
### ACTIONS ###
# DISRUPTIVE
allow Stop processing of one or more remaining phases
block Indicate that a rule wants to block
deny Block transaction with an error page
drop Close network connection
pass Do not block, go to the next rule
pause Pause for a period of time, then execute allow.
proxy Proxy request to a backend web server
redirect Redirect request to some other web server
# FLOW
chain Connect two or more rules into a single logical rule
skip Skip over one or more rules that follow
skipAfter Skip after the rule or marker with the provided ID
Others..
#METADATA, #VARIABLE, #LOGGING, #SPECIAL, #MISC

SSL Review part 1

Most of us use and rely on SSL everyday. The mathematical workings of the RSA [Rivest, Shamir, Adleman] algorithm are not overly complex but mapping everything back to what happens in reality requires detailed understanding. Skipping over the need for SSL (for confidential and authenticated exchange of a symmetric key over and insecure medium) I will review the mathematical workings then how they are applied in real world examples.

There are also details in previous posts – RSA1, RSA2

Mathematics 

Step
Components
1. public key – e
standard practice to choose: 65537 
2. random primes p,q
Let’s use:

579810099525248565010050509754571001027,

6989752565699505597485398979958574481969

3. key modulus – n
 n = pq:

4052729130775091849638047446256554071699019514021047339267026030072286291982163

4. φ(n) = (p – 1)(q – 1)
 φ(n):

4052729130775091849638047446256554071691449951355822585104530580582573146499168

5. find e that is co-prime with φ(n)
 already using a prime,which will be co- prime.. – e = 65537 (in binary 10000000000000001)
6. (ed) mod φ(n) = 1d = e–1 mod φ(n) –  Modular multiplicative inverseMore than one answer
using Extended Euclidean algorithm:

944402082567056818708092537028397604145319798848072425038015030084640082599681,

4997131213342148668346139983284951675836769750203895010142545610667213229098849,

..+ 2φ(n), +3φ(n))

7. private key – d
 de–1 mod φ(n):

944402082567056818708092537028397604145319798848072425038015030084640082599681

With a public key (e), a key modulus (n) and a private key (d) we can apply the RSA algorithm.

Message (mess) = 911

RSA encrypt -> mess ^ e mod n  = 911 ^65537 mod 4052729130775091849638047446256554071699019514021047339267026030072286291982163

RSA encrypted message (ciph) = 3095021178047041558314072884014000324030086129008597834642883051983162360819331

RSA decrypt -> ciph ^ d mod n = 3095021178047041558314072884014000324030086129008597834642883051983162360819331 ^ 944402082567056818708092537028397604145319798848072425038015030084640082599681 mod

4052729130775091849638047446256554071699019514021047339267026030072286291982163

= 911

How is that secure?

When Alice encrypts using Bob’s public key (e) along with the key modulus (n) the output is a protected cipher.

An eavesdropper does not know the private key so decryption is very difficult:

Attacker must solve:

(unknown val, x) ^ e mod n = ciph

x ^65537 mod 4052729130775091849638047446256554071699019514021047339267026030072286291982163 = 3095021178047041558314072884014000324030086129008597834642883051983162360819331

OR, easier – try to determine the private key:

The attacker knows e and n (which = pq). When we created the private key (step 6 above) we conducted:  e–1 mod φ(n) – Modular multiplicative inverse which is relatively fast for us to calculate.

The attacked does not know  e–1 mod φ(n) though. φ(n) = (p – 1)(q – 1). The attacker knows that n is a composite prime = pq (where p and q are both primes).

So… if the attacker can solve p * q = n (where they know n) then RSA is insecure.

Thankfully the process of Integer factorization is so much harder than the process of creating p,q,nφ(n), e and d that online business and confidentiality can be maintained to acceptable levels.

Threats to RSA

It would be extremely valuable to malicious individuals/groups and  (more importantly) intelligence organizations make large integer factorization efficient enough to break RSA.

However the theoretical aspects of RSA are not generally recognized as the main source of vulnerability

Week 3 – Introduction to R. Cont’

Coming back to R after closing, a session can be restored by simply running R in the workspace directory.

A history file can be specified via:

# recall your command history 
loadhistory(file="myfile") # default is ".Rhistory"

RData can also be saved and loaded via:

# save the workspace to the file .RData in the cwd 
save.image()

# save specific objects to a file
# if you don't specify the path, the cwd is assumed 
save(object list,file="myfile.RData")
# load a workspace into the current session
# if you don't specify the path, the cwd is assumed 
load("myfile.RData")

Describing data:

# show data files attached
ls()
# show dimensions of  a data object 'd'
dim(d)
#show structure of data object 'd'
str(d)
#summary of data 'd'
summary(d)

Subsets of data is a logical next step:

summary(subset(d, read <= 60))

Grouping data is also fairly intuitive:

by(d[, 7:11], d$prog, colMeans)
by(d[, 7:11], d$prog, summary)

Using histograms to plot variable distributions:

ggplot(d, aes(x = write)) + geom_histogram()
# Or kernel density plots
ggplot(d, aes(x = write)) + geom_density()
# Or boxplots showing the median, lower and upper quartiles and the full range
ggplot(d, aes(x = 1, y = math)) + geom_boxplot()

Lets look at some more ways to understand the data set:

# density plots by program type
ggplot(d, aes(x = write)) + geom_density() + facet_wrap(~prog)
# box plot of math scores for each teaching program
ggplot(d, aes(x = factor(prog), y = math)) + geom_boxplot()

Extending visualizations:

ggplot(melt(d[, 7:11]), aes(x = variable, y = value)) + geom_boxplot()
# break down by program:
ggplot(melt(d[, 6:11], id.vars = "prog"), aes(x = variable, y = value, fill = factor(prog))) +  geom_boxplot()

Analysis of categories can be conducted with frequency tables:

xtabs(~female, data = d)
xtabs(~race, data = d)
xtabs(~prog, data = d)
xtabs(~ses + schtyp, data = d)

Finally lets have a look at some bivatiate (pairwise) correlations. If ther is no missing data, cor function can be users, else use can remove items:

cor(d[, 7:11])
ggpairs(d[, 7:11])

 

Week 2 – Introduction to R.

Why use R?

R is an integrated suite of software facilities for data manipulation, calculation and graphical display. – http://cran.csiro.au/doc/manuals/r-release/R-intro.html

This is a valid question considering that most languages/frameworks, including CUDA have statistical analysis libraries built in. Hopefully running through some introductory exercises will reveal the benefits.

Associated GUI’s and extensions:

  • Weka – Specific for machine learning algorithms
  • R Commander – Data analysis GUI

Install on Ubuntu 12.04:

  1. sudo echo "deb http://cran.csiro.au/bin/linux/ubuntu precise/" >> /etc/apt/sources.list
    sudo apt-get update
    sudo apt-get install r-base

    Then to enter the R command line interface, $ R

    For starters, will run through an intro from UCLA: http://www.ats.ucla.edu/stat/r/seminars/intro.htm

    Within the R command line interface if a package is to be used it must first be installed:

    install.packages()
    • foreign – package to read data files from other stats packages
    • xlsx – package (requires Java to be installed, same architecture as your R version, also the rJava package and xlsxjars package)
    • reshape2 – package to easily melt data to long form
    • ggplot2 – package for elegant data visualization using the Grammar of Graphics
    • GGally – package for scatter plot matrices
    • vcd – package for visualizing and analyzing categorical data
    install.packages("xlsx")
    install.packages("reshape2")
    install.packages("ggplot2")
    install.packages("GGally")
    install.packages("vcd")

    Pre-requisites:

    sudo apt-get install openjdk-7-*
    sudo ln -s /usr/lib/jvm/java-7-openjdk-amd64/bin/java /etc/alternatives/java
    sudo R CMD javareconf

    Preparing session:

    After installing R and the packages needed for a task if these packages are needed in the current session they must be included:

    require(foreign)
    require(xlsx)

    After attaching all of the required packages to the current session, confirmation can be completed via:

    sessionInfo()

    R code can be entered into the command line directly or saved to a script which can be run inside a session using the ‘source’ function.

    Help can be attained using ? preceding a function name.

    Entering Data:

    R is most compatible with datasets stored as text files, ie: csv.

    Base R contains functions read.table and read.csv see the help files on these functions for many options.

    # comma separated values
    dat.csv <- read.csv("http://www.ats.ucla.edu/stat/data/hsb2.csv")
    # tab separated values
    dat.tab <- read.table("http://www.ats.ucla.edu/stat/data/hsb2.txt", header=TRUE, sep = "t")
    

    Datasets from other statistical analysis software can be imported using the foreign package:

    require(foreign)
    # SPSS files
    dat.spss <- read.spss("http://www.ats.ucla.edu/stat/data/hsb2.sav", to.data.frame=TRUE)
    # Stata files
    dat.dta <- read.dta("http://www.ats.ucla.edu/stat/data/hsb2.dta")

    If converting excel spreadsheets to CSV is too much of a hassle the xlxs package we imported will do the job:

    # these two steps only needed to read excel files from the internet
    f <- tempfile("hsb2", fileext=".xls")
    download.file("http://www.ats.ucla.edu/stat/data/hsb2.xls", f, mode="wb")
    dat.xls <- read.xlsx(f, sheetIndex=1)

    Viewing Data:

    # first few rows
    head(dat.csv)
    # last few rows
    tail(dat.csv)
    # variable names
    colnames(dat.csv)
    # pop-up view of entire data set (uncomment to run)
    View(dat.csv)
    

    Datasets that have been read in are stored as data frames which have a matrix structure. The most common method of indexing is object[row,column] but many others are available.

    # single cell value
    dat.csv[2, 3]
    # omitting row value implies all rows; here all rows in column 3
    dat.csv[, 3]
    # omitting column values implies all columns; here all columns in row 2
    dat.csv[2, ]
    # can also use ranges - rows 2 and 3, columns 2 and 3
    dat.csv[2:3, 2:3]

    Variables can also be accessed via their names:

    # get first 10 rows of variable female using two methods
    dat.csv[1:10, "female"]
    dat.csv$female[1:10]

    The c function is used to combine values of common type together to form a vector:

    # get column 1 for rows 1, 3 and 5
    dat.csv[c(1, 3, 5), 1]
    ## [1]  70  86 172
    # get row 1 values for variables female, prog and socst
    dat.csv[1, c("female", "prog", "socst")]
    ##   female prog socst
    ## 1      0    1    57

    Creating colnames:

    colnames(dat.csv) <- c("ID", "Sex", "Ethnicity", "SES", "SchoolType", "Program", 
        "Reading", "Writing", "Math", "Science", "SocialStudies")
    
    # to change one variable name, just use indexing
    colnames(dat.csv)[1] <- "ID2"

    Saving data:

    #write.csv(dat.csv, file = "path/to/save/filename.csv")
    #write.table(dat.csv, file = "path/to/save/filename.txt", sep = "t", na=".")
    #write.dta(dat.csv, file = "path/to/save/filename.dta")
    #write.xlsx(dat.csv, file = "path/to/save/filename.xlsx", sheetName="hsb2")
    # save to binary R format (can save multiple datasets and R objects)
    #save(dat.csv, dat.dta, dat.spss, dat.txt, file = "path/to/save/filename.RData")
    #change workspace directory
    setwd("/home/a/Desktop/R/testspace1")

downloading a youtube playlist and converting to mp3 (linux)

** A lot of music on youtube (that I listen to) is unsigned, if the music is released, support the artists and buy the tracks at beatport/itunes etc **

Made a 5 min script that does all the steps needed to download a youtube playlist and convert it to mp3. It is not perfectly efficient and could be significantly improved by checking if the song exists in that dir already.

This script is simply joining together:

Requirements: youtube-dl, ffmpeg, Python version 2.6, 2.7, or 3.3+ , libmp3lame0

Steps to get these on Ubuntu:

apt-get install ubuntu-restricted-extras

apt-get install libmp3lame0

apt-get install youtube-dl

apt-get install ffmpeg

 

Usage: sh GetPlaylist.sh [youtube playlist URL] [output_directory]

Example:

sh ./GetYoutubePlaylist.sh http://www.youtube.com/playlist?list=PL702CAF4AD2AED35B /home/user1/Music/youtube/

To get the youtube playlist URL view the playlist by clicking on its name, no videos will be playing, then click on share and the url will be highlighted:

youtube-playlist
click on share and the URL will appear

note: any other .flv/.mp4 files in that directory will be converted to mp3 and deleted.

The script:

# Simple script to dl and convert youtube playlists - MarkC, 2013
# Usage: sh GetYoutubePlaylist.sh [youtube playlist addess] [output directory]
# To get the youtube playlist address, view the playlist then click on share, copy that URL
# example: sh GetYouTubePlaylist.sh http://www.youtube.com/playlist?list=PL702CAF4AD2AED35B /home/lp1/Music/youtube
# ytdl - abritrary extension name
outputFileName="/%(playlist)s/%(title)s"
youtube-dl -o "$2%(playlist)s/%(title)s.ytdl" --max-quality url $1
find $2 -type f -name "*.ytdl" -exec avconv -i '{}' '{}'.mp3 ;
find $2 -name *.ytdl -exec rm {} +

 

Introduction to Parallel Programming – Week 5 Part 2

Week 5 -> Optimizing GPU Programs (continued)

Goal: Maximize useful computation/second

After walking through APOD [Analyze, parallelize, optimize, deploy] the lecture turned to memory bandwidth. Using the CUDA utility deviceQuery to calculate memory bandwidth using memory clock rate and memory bus width.

After determining the maximum theoretical bandwidth at 40 Gb/s practical goals were set:

  • 40-60% -> OK
  • 60-75% -> good
  • >75% -> excellent
Using 12.5 GB/s of memory bandwidth is under utilizing!
Using 12.5 GB/s of memory bandwidth is under utilizing!

The algorithm being analyzed is a transpose. When DRAM utilization is low the first guess should be lack of coalescing.

In our example code we find that the reading of memory is well coalesced.. but the writing phase is strided by N elements (1024). This was described as ‘bad’.

So considering that most GPU codes are memory limited checking memory bandwidth utilization is very important.

Enter a tool – nSightm nvpp [nvidia visual profiler]  – confirms that the write to memory operations are utilizing very little memory bandwidth whilst the read  operations are at 100% utilization.

The solution for this algorithm is ’tiling’. Tiling utilizes shared memory, taking a tile of the input copying and transposing into output. The code for this solution can be found here: https://github.com/udacity/cs344/blob/master/Unit5%20Code%20Snippets/transpose.cu

Occupancy – Each SM has a limited number of:

  • Thread blocks -> 8
  • Threadss -> 1536/2048
  • registers for all threads -> 65536
  • bytes of shared memory 16K-48K

Maximizing number of threads on the SM [streaming multi processor] will maximize occupancy. The limits for specific hardware can be found via deviceQuery.

The transpose code was further optimized, better versions can be seen in the link above.

Shared memory bank conflicts ->organized into banks and depending on how threads access memory in tile, replays of shared memory accesses can occur. (ie: striping shared memory usage across banks).

Referring back to the goal of maximizing effective computation we have just address one factor:

  • minimize time waiting at barriers

The next to be address was:

  • minimize thread divergence

Some important definitions:

  • warp – set of threads that execute the same instruction at a time
  • SIMD – Single instruction, multiple data (CPU
  • SIMT – Single instruction, multiple thread

Thread divergence is can result in up to 32x slower code. Warps on nvidia hardware have 32 threads which apply single instruction to multiple threads.

Next topic was Streams, launching kernels in separate streams allows for concurrent execution:

cuda_streams
Benefits of specifying CUDA streams

To create streams:

cudaSteam_t s1;

cudaStreamCreate(&s1);

cudaStreamDestroy(s1);

asynchronous memory transfer  – cudaMemcpyAsync – called on pinned memory.

Week 5 lectures

Week 5 code – failed to get shared memory working even with atomicAdd :(