Hypothesis Testing

A very important part of data science/statistics is hypothesis testing. We need this testing to assess likelihoods of certain possibilities according to the data set we have and the problem we are trying to solve. Let’s define what hypothesis testing means.

—————————–

Hypothesis testing helps to evaluate two disjoint or mutually exclusive events about any given population in order to find out which of those events hold true. There are few steps that need to be followed to perform this testing and they are as follows:

a. State the null hypothesis (H0)

b. Then alternate hypothesis (Ha)

c. Set α. A contingency table could be built to see the status of hypothesis claimed and figure out the Type-I & II as we need to set α before the experiment.

d. Collect the data

e. Next step is to use some kind of statistic like T-Statistic or for categorical data, we can use f-statistic.

f. Find the accept-null and reject-null hypothesis regions on the graph.

g. At the end, draw conclusions about null hypothesis. “If the p-value obtained from the ANOVA is less than α, then Reject H0 and Accept Ha.”

—————————-

Let’s learn about some of the very essential terminologies involved in this type of testing and others related to it.

  1. Confidence Interval (C.I): We calculate the means of samples from the population. We need range of values that will encompass all the possible point estimates. That is what confidence interval is for. In order to build confidence interval, we need random sample and the sampling distribution should be approximately or nearly normal. While α (level of significance) reflects how much risk we are willing to take in our results, confidence interval is: I = 1 – αPoint estimates: When we calculate the mean of sample, it is an unbiased estimate for mean of population. It can be considered as a point of estimate for a value. C.I is just the range for all these point estimates.
  2. Variance: Average of squared differences from the mean. At first you need to calculate the mean, then for each number in the sample, calculate the difference between mean and the number, square the result. At the end, calculate the average of those squared differences or results.
  3. Covariance: The measure of change between two variables, how change in one variable is associated with the change in the other variable.
  4. ANOVA: Analysis of variance is nothing but a collection of various statistical models used to figure the differences of means among or between the groups in a dataset. It is a test that provides the confirmation whether the means of groups are equal. This test is useful to test the statistical significance for 3 or more means of variables. To perform ANOVA, we need the response variable to be continuous, and at least one categorical variable. Of course, it does need to have a normal distribution. Null hypothesis is that all the means are equal whereas the alternate hypothesis claims at least one of the mean is not equal.
  5. Standard deviation: square root of variance. Standard deviation (σ) is the measure to check how far spread the numbers are.
  6. P-value: The p-value is the probability of obtaining results that will let null hypothesis be true. There are certain ways to interpret the p-values we get, when p-value is less than α (level of significance), the model is statistically significant. The following table explains when is null hypothesis accepted or rejected:
Accept H0 Reject H0
H0   is true True Type I error
H0   is false Type II error Correct decision

So what’s Type I & II errors?

Type I error: incorrect rejection of H0, no effect of sample size as it is already set, increases with the number of end points.

Type II error: incorrectly accepting H0

When do we reject null hypothesis?

When the p-value is less than the chosen significance level, null hypothesis is rejected. This is when we say the result is statistically significant.

The steps of hypothesis testing, we mentioned about choosing statistical test, so now the question arises, how do we decide which test we want to use? We can answer to our own questions by asking the following questions while we analyze our data:

-Is the data categorical or quantitative (measurements, counts, scales)?

-How many groups are there in the data? For example, two sample data for one group of people. When we have two samples of data and it is categorical, we can use chi-squared test.

-What’s the hypothesis meant to do, as in comparison of data or finding a relationship (risk factor)?

6. Normal distribution: Data can be spread out in different ways, like, more towards the left on the axis, more towards right or all in the center forming a bell curve, in the graph. This leads to a normal distribution without a bias to left or right. Normal distribution has mean = median = mode, a symmetry about the center, 50% values less than mean and 50% greater than mean. When we know standard deviation, we can say that any value is likely to be in 1 std dev (68/100), very likely to be within 2 std dev (95/100) or almost certainly to be within 3 std dev (997/1000).

7. Standard normal distribution: This helps to make decisions about our data. To standardize, subtract the value from mean and then divide by std dev. We get the z-score and this is called standardizing normal distributions.

8. Z-score =   value – mean(mu)/std.dev , x is value, z is z-score and  is standard deviation.

9. Mean(sample) = mean(population), variance(sample) < variance(population), sample-size: n > 30 is sufficiently large.

There are two kinds of statistical tests:

a. Parametric test: Parametric tests work really well with skewed and non-normal distribution of data. This test is more preferable because they are more likely to detect a difference, and less likely than non-parametric tests to make a Type II error.

i. Independent-samples t test

ii. Paired-samples test

iii. One-way ANOVA: used when there are 3 or more samples

Assumptions made: random independent samples, interval or ratio level of measurement, normal distribution, no outliers, homogeneity of variance, sample sizes larger than minimum for many non-parametric tests.

b. Non-parametric test: This test works well when you have small sample size, irremovable outliers, ordinal or ranked data, and median represents the area of study better than anything. This test surely has less statistical power.

i. Mann-Whitney test: assumes two samples have same shape

ii.  Wilcoxon signed-rank test: assumes symmetric distribution

iii. Kruskal-Walis test: assumes same shape and equal variance.

——————

References:

Hypothesis Testing: https://onlinecourses.science.psu.edu/stat502/node/139

Cliffsnotes. https://www.cliffsnotes.com/study-guides/statistics/principles-of-testing/point-estimates-and-confidence-intervals

SQL is necessity of Data Science

/* Basic SQL statements for celebs table which has id (type-integer), name(type-text), and age(type-integer)
of the celebs
Reference: Codecademy course (in progress)
*/
— Create celebs table
CREATE TABLE celebs (id INTEGER, name TEXT, age INTEGER);

— Select all the data from the table
SELECT * FROM celebs;

— Insert new entries in celebs table
INSERT INTO celebs (id, name, age)
VALUES (1, ‘Taylor Swift’, 26);
INSERT INTO celebs (id, name, age)
VALUES (2, ‘Enrique’, 33);
INSERT INTO celebs (id, name, age)
VALUES (3, ‘Joe Jonas’, 26);

— Edit or Update any row in the table
UPDATE celebs
SET age = 23
WHERE id = 1;

— Add a new column in the table
ALTER TABLE celebs ADD COLUMN
twitter_handle TEXT;

–Now adding value to the newly created column
UPDATE celebs
SET twitter_handle =
‘@taylorswift13’
WHERE id = 1;

— Before adding the value to the new column, it had NULL as its value which normally represents missing or unknown data

— Delete all the rows that still has NULL value
DELETE FROM celebs WHERE
twitter_handle IS NULL;

/*
Here comes the Queries. Codeacademy used movies database that had name, genre, imdb_rating etc.
*/

–Let’s select distinct genres, that means no duplicate values from the database
SELECT DISTINCT genre FROM movies;

–Filtering of data using different queries and condition
— Like is an operator that can be used when you want to compare two similar values

SELECT * FROM movies
WHERE name LIKE ‘%man%’;
— This returns all the columns from the database where the name of the movie contains the text man in it
— % matches 0 or more missing letters in the pattern
— anything before % any text matches everything that starts with that string
— anything after % any text matches everything that ends with that string

 

The ‘OOP’ moments with Python

NOTE: The code snippets were written in Notepad++ so sadly the tabs (4 spaces) are missing!

# So what is OOP with Python like?

#let’s start with making classes, then objects along with them

class Customer(object):
#The above line is how you create a class and pass object to it

def __init__(self, name, balance = 0.0):
# This is how you create the instance function for the class you created
# self is a just an instance of class customer.
# __init__ is used to initialize attributes of the object
self.name = name
self.balance = balance
# The above two lines is how you instantiate the class with the values in the method

def deposit(self, amount):
self.balance += amount
return self.balance

custCall = Customer(‘Jake’) # Calling the class by passing customer’s name as object
#Passing attribute to an instance object

print custCall.deposit(4000) # This will call the instance method named deposit
# This was the regular way of initializing attributes but in Python you can also do the same dynamically.
# Reference: http://zetcode.com/lang/python/oop/

class Dynamic:
pass # You pass onto the next line since you don’t have any objects created here, yet.

dy = Dynamic() # Create instance of the class
dy.name = “Dynamic” #Passing value to the instance attribute
print dy.name

#Output: Dynamic
#########################
# Note on Encapsulation and Abstraction
# Well these could be used as synonyms. Data hiding is their main purpose
# When we talk about encapsulation, we always need methods inside the class structure
# Methods are necessary in distributing the work in our code
# We need to access the data from those instance methods, thus we need get() and set()
# In more detail, http://tuxlabs.com/?p=207
# when a method’s name starts with _ it is a protected method
# when a method’s name starts with __ it is a private method, otherwise public
#########################

# Inheritance
# You have a class defined already and you make a new one and derive features from the already defined one
# is called inheritance
# A very famous example to explain inheritance is of animals

class Animal:
def __init__(self):
print “Animal kingdom”
def eat(self):
print “Eat”
def talk(self):
pass
class Cat(Animal): # class cat inherits the features from class Animal
def __init__(self):
Animal.__init__(self)
print “Dog class is created”
def talk(self):
print “meow meow”
c = Cat(“chicky”)
c.talk()

# Multiple inheritance is also possible in Python. Simply put class names
# inside the super class and separate them by comma.
#########################

# Polymorphism
# We looked at inheritance where class cat inherited features from class animal
# Polymorphism property says that class cat could do things differently from class animal
# basically it does not have to inherit each and every feature exactly.
# This being said, polymorphism is commonly used when inheritance is and
# Python extensively uses polymorphism in its built-in types

# Extending the above animal example
class Dog(Animal):
def talk(self):
print “Woof Woof”

d = Dog(“tommy”)
d.talk()

# Both classes Cat and Dog inherits the method of talk() from class Animal
# but will get different output from each other.
# This explains polymorphism since we used same method but differently

Hackerrank is life’s hack

Some of the interesting problems in Python domain at hackerrank that I solved

# Hackerrank interesting problems

  1. Hackerrank interesting problem with using print function to print integers in one line:
    Task
    Read an integer NN. Without using any string methods, try to print the following:

1,2,3…..N1,2,3…..N.  Note that “…..” represents the values in between.

Input Format
The first line contains an integer NN.

Output Format
Output the answer as explained in the task.

Code:
print (*range(1, int(input()) + 1), sep = ”)
# whenever you add * in front of expression, that expression becomes iterable

———————————–

2. You are given a string SS.
The string contains only lowercase English alphabet characters.

Your task is to find the top three most common characters in the string SS.

Input Format

A single line of input containing the string SS.

Constraints

3<len(S)<1043<len(S)<104
Output Format

Print the three most common characters along with their occurrence count each on a separate line.
Sort output in descending order of occurrence count.
If the occurrence count is the same, sort the characters in ascending order.

Code:
import operator

S = input() #get the input string from the user

dictLetterCount = {} # dictionary for letters and their count

for c in S:
if c in dictLetterCount:
dictLetterCount[c] += 1
else:
dictLetterCount[c] = 1
dictLetterCountSorted = sorted(dictLetterCount.items(), key = lambda kvPair:(-kvPair[1], kvPair[0])) #sorts by value at first then by keys
for k, v in dictLetterCountSorted[:3]:
print (‘{} {}’.format(k, v)) # to decompose the tuples in dictLetterCountSorted

#sorted(dictLetterCount.items()) # sorts by keys #outputs the list

#sorted(dictLetterCount.items(), key = operator.itemgetter(1, 0)) #itemgetter gets the 2nd item from the list
#dictByValues = sorted(dictLetterCount.items(), key = operator.itemgetter(1), reverse=True) #sorting by values
# stable sort
#dictByValuesByKeys.sort(key = operator.itemgetter(0))
#dictByValuesByKeys = sorted(dictByValues, key = operator.itemgetter(0))# not using .items() since it becomes a list of tuples

#worst case: o(nlogn) if all the letters were unique

3.Task: Initialize your list (L = []) and follow the NN commands given over NN lines.

Each command will be 11 of the 88 commands given above. The extend(LL) method will not be used. Each command will have its own value(s) separated by a space.

Input Format

The first line contains an integer, NN (the number of commands).
The NN subsequent lines each contain one of the 88 commands described above.

Code: 

N = int(input()) # number of commands
L = []

dictOfCommands= {‘append’:getattr(list,’append’), ‘remove’:getattr(list,’remove’),
‘insert’:getattr(list,’insert’), ‘pop’:getattr(list,’pop’),
‘index’:getattr(list,’index’), ‘count’:getattr(list,’count’),
‘sort’:getattr(list,’sort’), ‘reverse’:getattr(list,’reverse’)}

for _ in range(N):
command = input().split(‘ ‘)
if command[0] == ‘print’:
print(L)
else:
        dictOfCommands[command[0]](L,*map(int,command[1:]))

#decomposition of the list, mapping the integers in the list to convert into int, hence used map(), used getattr() that is called from this dictionary and the commands are executed

”’
a = [1,2,3]
def myfun(a,b,c):
…
myfun(*a)

#Example for what is going on in the line of code marked red

list = [‘insert’,’1′,’7′]
params = list[1:]
intparams = map(int,params)
insert(L,*intparams)
”’

R is fun

##### Interview for data science : https://www.dezyre.com/article/100-data-science-in-r-interview-questions-and-answers-for-2016/187?utm_content=buffer49cf0&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer
> mean(as.numeric(examDataset$V2))
[1] 15.5

>sd(as.numeric(examDataset$V2))
[1] 8.803408

d <- density(as.numeric(examDf$V2))
> plot(d)
plotting normal distribution

examDf$V4 <- as.numeric(as.character(examDf$V4)
densfuncr1ppp <- examDf$V4
plot(densfuncr1ppp, dnorm(densfuncr1ppp, meanr1ppp, sdr1ppp), type = “h”)
# Cleaning data in R

1. read csv file in R
2. Convert the csv file into a data frame or matrix, as per needed
3. Use complete.cases() as follows:
a. Make a new object to save the cleaned subset of your main data:
newObj <- oldDF[complete.cases(oldDF),]
str(newObj)

> # Output for above is:
‘data.frame’: 23 obs. of 8 variables:
$ Survey.Scale.: Factor w/ 11 levels “”,”2010″,”2011″,..: 2 3 4 5 6 2 3 4 5 6 …
$ X0 : int 1 0 1 1 2 0 0 0 0 1 …
$ X1 : int 0 0 1 2 3 0 0 0 1 1 …
$ X2 : int 2 2 1 6 5 0 0 1 1 2 …
$ X3 : int 14 14 8 12 15 2 2 4 3 4 …
$ X4 : int 22 20 34 34 44 6 6 11 12 22 …
$ X5 : int 11 14 15 45 56 2 2 14 33 60 …
$ Sample.Size : int 50 50 60 100 125 10 10 30 50 90 …

# summary() does not work if you have character variables in the data frame, instead
# you can use aggregate()
# str() :extremely useful to get the structure of dataset, to see numerical vs categorical fields in the dataset
# describe(): comes handy to see the number of NAs and several other details about the dataset. You will need HMisc package to use describe()

# http://ww2.coastal.edu/kingw/statistics/R-tutorials/simplelinear.html (Following code example is taken from this website)
# plot() : 2 variables can create a scatter plot for you, where the first variable provided is for
#horizontal axis and the second one is for verticle

#plot() can be used as a formula interface where the response variable should appear before the tilde
#and the horizontal axis variable should come after the tilde. Example: (2 ways to do this)

with(cats, plot(Hwt ~ Bwt)) #or
plot(Hwt ~ Bwt, data = cats)

# This creates a scatter plot, now if we want to add a regression line to get a clear visualization of
#linearity in the plot, we use:

abline(lm(Hwt ~ Bwt), data = cats) # specifying the dataframe we are using is necessary otherwise it gives the following error:

# Error in eval(expr, envir, enclos) : object ‘Hwt’ not found

# To analyze the strength of correlation of our linearity, we use Pearson’s R correlation coefficient
# stronger the relation between the two variables, closer will be the Pearson coefficient either to +1 or -1 (positive & negative correlation resp.)

#Are there guidelines to interpreting Pearson’s correlation coefficient?
#Yes, the following guidelines have been proposed:

# Coefficient, r
#Strength of Association Positive Negative
#Small .1 to .3 -0.1 to -0.3
#Medium .3 to .5 -0.3 to -0.5
#Large .5 to 1.0 -0.5 to -1.0
#Remember that these values are guidelines and whether an association is strong or not will also depend on what you are measuring.
# [https://statistics.laerd.com/statistical-guides/pearson-correlation-coefficient-statistical-guide.php]
> with(cats, cor(Bwt, Hwt)) #one way to find the correlation
[1] 0.8041274
> cor.test(~Bwt + Hwt, data = cats) # uses t-test with 95% confidence level. A different conf level can be provided if need be

Pearson’s product-moment
correlation

data: Bwt and Hwt
t = 16.1194, df = 142, p-value <
2.2e-16
alternative hypothesis: true correlation is not equal to 0
95 percent confidence interval:
0.7375682 0.8552122
sample estimates:
cor
0.8041274

# This analysis indicates that the linear relationship between the body weight and heart weight of cats is strongly significant. The heart weight
# increases with the increase in the cat’s heart weight

# Since the data set has data for both male and female cats, we can subset for finding the correlation analysis of female or male cats

Python Learning day by day

This post will keep getting updated either in the same blog post or will get published in different versions.

# Notes on Python’s pandas and everything else

Pandas have dataframes. With the dataframes, we can have several operations.

1. Read into csv file
2. Lookup the first five rows of the dataset by
first_rows = dataset.head()
We can add in an integer to display any number <= dataset size into head()

3. We can see how the dataset is setup/seeing the dimensions of dataset by using:
dimensions = dataset.shape
num_rows = dimensions[0] # to see rows
num_cols = dimensions[1] # to see columns
This returns the tuple

4. Pandas have series object that returns the row/column labels with the values corresponding to them instead of
returning a list of row/column values.

5. loc[] is to select any row from the dataset

6. Datatypes in pandas are a little different from the basic programming language datatypes we have been studying
object is equivalent to strings datatypes, then we have int, float, datetime, bool

7. Pandas can do a lot with extracting columns from the dataset like selecting multiple columns from the dataframe
and it returns the columns in the order they are passed even though they are not situated together in the dataset:
colList = dataset[[“col1”, “col2”]]
———————————————–
1. Generators and iterables have two differences among them: (for more details: https://pythontips.com/2013/09/29/the-python-yield-keyword-explained/)
Generators use ()
Iterables use []

Generators can be traversed once
Iterables can be traversed as many times you want

Strings, lists, tuples are all iterables and so are generators.

2. Yield works similar to return except the function returns a generator

3. Yield and generator combination works efficiently in tasks involving sequential concept like generating a fibonacci series:
(http://pythoncentral.io/python-generators-and-yield-keyword/)
Example:
def fibonacci(n):
curr = 1
prev = 0
counter = 0
while counter < n:
yield curr
prev, curr = curr, prev + curr
counter += 1
yield and xrange can be treated similarly since both of them generates the value as the loop progresses rather than creating the list beforehand.

———————————————–
Python data structure complexities: https://wiki.python.org/moin/TimeComplexity

1. Linked lists: To delete any element from a linked list and replace it with a new one, we need to make sure we add reference of the next element in this new element otherwise we may lose the reference for it.

References: https://developers.google.com/edu/python/lists#list-methods

 

The magic of getattr():

Came across this brilliant method which can take in the attributes for any object you create. The example is as follows:

a = ‘sort’ #assigning the sort method to a variable

myList = [2,4,5,1,99] #creating a list of 5 integers to sort in ascending order

myFunc = getattr (list, a) # Now myFunc gets the list we created and applies the sort function that we stored in the variable ‘a’.

myList #we now check the output

Output: [1, 2, 4, 5, 99]

# Slice it up with Python Slices in lists

#palindrome:
def palindromeCheck(number):
return number == number[::-1]

palindromeCheck(‘141’)
#output: True
palindromeCheck(‘154’)
# output: False

#Isn’t this brilliant how checking up a palindrome in Python is about 2 lines of code.

#power of slicing in the following image

slice