Introduction to Clustering using R

We all have heard of Big Data a lot! Want to try basic clustering techniques on it?

Here we go. Before beginning into the coding , lets familiarize ourselves with some basic terms and platforms to do this thing.

1. What data are we studying?
For this purpose we would be studying Twitter, there are well equipped APIs to study the tweets that too in varied languages, eg, twitteR in R (stastical tool), tweetPy in python.

2. Tools we would be using: R
Get yourself quickly familiarized with R by following these links:

http://www.r-bloggers.com/how-to-learn-r-a-flow-chart/
http://tryr.codeschool.com/

And while searching for the relevant links, I have found this course for you, you may want to attend:

https://www.coursera.org/course/rprog


3. What we aim to do?
Collect the list of friends and followers for any one person's twitter account.
We would make use of user class of twitteR for the same.
It has following fields:

• name:Name of the user
• screenName:Screen name of the user
• id:ID value for this user
• lastStatus:Last status update for the user
• description:User’s description
• statusesCount:Number of status updates this user has had
• followersCount:Number of followers for this user
• favoritesCount:Number of favorites for this user
• friendsCount:Number of followees for this user
• url:A URL associated with this user
• created:When this user was created
• protected:Whether or not this user is protected
• verified:Whether or not this user is verified
• location:Location of the user
• listedCount:The number of times this user appears in public lists
• followRequestSent:If authenticated via OAuth, will be TRUE if you’ve sent a friend request to this user
• profileImageUrl:URL of the user’s profile image, if one exists

and several methods. To look into what all things twitteR API offers, check out this: http://cran.r-project.org/web/packages/twitteR/twitteR.pdf 

So, what we intend to do on the basis of this information is quite simple (and not that useful :P ) but may serve as a good introduction to clustering.

We accumulate a set of users on twitter, by fetching a twitter user's friends and followers as User class objects . We take the union of these users. Now we plot a graph friends vs followers, where each dot is a 2 D / 3 D entity containing friends' count, followers' count and/ or statuses' count.

We use inbuilt K-means clustering algorithm in RStudio to do the clustering on this data.
Pondering what is K-means? Check : https://sites.google.com/site/dataclusteringalgorithms/k-means-clustering-algorithm
or several explanatory videos available online.

So basically, here we have a feature vector of 3 features per user:
1. friend's count
2. follower's count
3. statuses'  count

On the basis of similarity measure, we cluster the data points together on the basis of similarity wrt to these 3 features.

***Check out what similarity kmeans internally uses in R and answer in comments if you find one, I guess it is cosine similarity. ***

These clusters can be further colored to mark density.

Follow the code given in this link to do it by yourself:

http://rstudio-pubs-static.s3.amazonaws.com/5983_af66eca6775f4528a72b8e243a6ecf2d.html


I used and tweaked the code a bit to do kmeans clustering on "modi" as user and used 2 features : friends count and followers count for first and
3 features: friends count , followers count and statuses' count for the next, this is what we attained:

  

 2 Features

                                                 3 Features


PS: It is a 2D graph still but the clusters have been redefined, I put k=5 for this, where k stands as the number of clusters we wish to generate using the kmeans algorithm.


So that's it for now. I hope it would have been interesting and useful.
There exists lots of exciting relevant stuff online, keep checking!

See you in next post, till then
Keep Hacking! :)

Diving Deep | Android

So here I am building my weather app on android studio, that would essentially fetch weather forecast for upcoming days. (covers lessons 1-3)

Check out:
https://github.com/udacity/Sunshine-Version-2

for more details on code we would be deliberating upon in this post. The post essentially doesn't deals with the whole details, it gives an idea on what kind of methods exist and their purpose and how it is all connected in an android app. Budding developers may find it a bit useful.

Since I have come a long distance till now, lets quickly sum up the way it is working as of now:

I have four main java classes till now:

1. MainActivity

2. Forecast Fragment

3. Detail Activity

4. Settings Activity

Let's peek into and see what these files are actually doing:

1. MainActivity: This extends ActionBarActivity
It has some basic overridden methods like:
(i) protected void onCreate(Bundle savedInstanceState)
It sets the content view to some layout XML using setContentView method.It sets the view to activity_main XML, it just has a framelayout component named container.

Then we have, onCreateOptionsMenu, this essentially inflates the main xml of menu. It has the xmls adding items like settings and refresh to it.

Then we also have one method named onOptionsItemSelected which deals with events related to selecting something from the menu bar. Comparisons like item.getItemId()==R.id.action_settings are done inside it.


2. ForecastFragment : One of the lengthiest.
It extends Fragment Class.
A Fragment is a piece of an application's user interface or behaviour that can be placed in an Activity.
In its core, it represents a particular operation or interface that is running within a larger Activity. A Fragment is closely tied to the Activity it is in, and can not be used apart from one. Though Fragment defines its own life cycle, that life cycle is dependent on its activity: if the activity is stopped, no fragments inside of it can be started; when the activity is destroyed, all fragments will be destroyed.

What does it particularly deals with?

Well, that is the list of weather forecasts we wanna display. To populate the list we need to have data in some form. To do the entire thong, we need to have ArrayAdapter

  Then we have all the overridden methods:
(i)  public void onCreate(Bundle savedInstanceState)
This just setOptionsMenu true so as to detect the presence of Menu Bar for the app.

(ii) public void onCreateOptionsMenu(Menu menu, MenuInflater inflater)
This inflates the menu view having refresh feature.

(iii) public boolean onOptionsItemSelected(MenuItem item)
It features the utility that is what will happen when refresh is clicked.
We used Shared Preferences and corresponding preference manager to generalize the input for city in which we need to check weather.
 We also created the object of async class FetchWeatherClass to fetch weather from the API in background.

(iv) public View onCreateView(LayoutInflater inflater, ViewGroup container,
                             Bundle savedInstanceState)
It inflates the required layouts and xmls
Creates array list and adds ArrayAdapter to it.
Components of ArrayAdapter :
1. The current context (this activity)
2. The name of the layout ID.
3. The ID of the textview to populate.
4. The array list i.e. weekForecast in this case.


We use listView object to display the list of weathers fetched. This list View is attached with the ArrayAdapter. A setOnItemClick listener is also attached to it so that on clicking any list item a different screen (activity) may come up using Intent.

Syntax for Intent is worth a deep glance:
Here it goes:

listView.setOnItemClickListener(new AdapterView.OnItemClickListener() {

            @Override
            public void onItemClick(AdapterView<?> adapterView, View view, int position, long l) {
                String forecast = mForecastAdapter.getItem(position);
                Intent intent = new Intent(getActivity(), DetailActivity.class)
                        .putExtra(Intent.EXTRA_TEXT, forecast);
                startActivity(intent);
            }
        });

We also have an inner class in this:
public class FetchWeatherTask extends AsyncTask<String, Void, String[]>
 This is essentially to retrieve and parse JSON data retrieved from the API into string array.
The doInBackground method essentially helps in URL construction and http connection to the API.
onPostExecute does the required add ons/ changes to the adapter to reflect the fetched data.

3. Detail Activity:
 Quite simple class. It extends ActionBarActivity too as the main activity does and also has an inner fragment class as a placeholder
The onCreate, onCreateOptionsMenu and onOptionsItemSelected methods have the same functionality as they are in the main activity apart from the required changes related to fragment.


The inner class  DetailFragment with the significant onCreateView method holds some significance , making canges to the displayed text view in accordance with the intent object it has received.

4. Settings Activity:
This extends the PreferenceActivity   class and implements the interface: Preference.OnPreferenceChangeListener
It makes use of xml file namely pref_general.
We do addPreferencesFromResource(R.xml.pref_general);
and  bindPreferenceSummaryToValue(findPreference(getString(R.string.pref_location_key)));

The second method Attaches a listener so the summary is always updated with the preference value. Also fires the listener once, to initialize the summary (so it shows up before the value is changed.)

Third method is : public boolean onPreferenceChange(Preference preference, Object value)
For list preferences, look up the correct display value in the preference's 'entries' list (since they have separate labels/values).
For other preferences, set the summary to the value's simple string representation.


Please note corresponding changes are being done in AndroidManifest file parallely.


So, that's all for now. Hope this post proves useful in some way. I just experimented on the ways an app's actual functionality can be understood from the point of view of a beginner.
Feel free to drop suggestions below.

Have a great day!

Keep hacking till then! :)