How would you make an HTML Parser?

Hola folks,

Here we will be walking through how html is being parsed by a widely popular parser: Angle Sharp
You can find it here !
This is just a walkthrough and gives an idea on the breadth of issues  one has to deal with while designing a parser.

Let's sneak peek into what kind of data structures are used and how is the code structured.

It all begins with this line:

var parser = new HtmlParser();

We have folowing variations in constructing the HtmlParser object:
   
  public HtmlParser()
            : this(Configuration.Default)
        {
        }

        /// <summary>
        /// Creates a new parser with the custom options.
        /// </summary>
        /// <param name="options">The options to use.</param>
        public HtmlParser(HtmlParserOptions options)
            : this(options, Configuration.Default)
        {
        }

        /// <summary>
        /// Creates a new parser with the custom configuration.
        /// </summary>
        /// <param name="configuration">The configuration to use.</param>
        public HtmlParser(IConfiguration configuration)
            : this(new HtmlParserOptions { IsScripting = configuration.IsScripting() }, configuration)
        {
        }

        /// <summary>
        /// Creates a new parser with the custom options and configuration.
        /// </summary>
        /// <param name="options">The options to use.</param>
        /// <param name="configuration">The configuration to use.</param>
        public HtmlParser(HtmlParserOptions options, IConfiguration configuration)
            : this(options, BrowsingContext.New(configuration))
        {
        }

        /// <summary>
        /// Creates a new parser with the custom options and the given context.
        /// </summary>
        /// <param name="options">The options to use.</param>
        /// <param name="context">The context to use.</param>
        public HtmlParser(HtmlParserOptions options, IBrowsingContext context)
        {
            _options = options;
            _context = context;
        }



You can find three things here:
1. Configuration:This is an interesting thing, opening it up we find an array of associated standard services like :
            Factory.HtmlElements,
            Factory.MathElements,
            Factory.SvgElements,
            Factory.Events,
            Factory.AttributeSelector,
            Factory.InputTypes,
            Factory.LinkRelations,
            Factory.MediaFeatures,
            Factory.Properties,
            Factory.PseudoClassSelector,
            Factory.PseudoElementSelector,
            Factory.Document,
            Factory.BrowsingContext
            Factory.Service
This can help us create customized configuration for contexts. Wondering what is Factory?
It is a static class bundling available factories. Factories are mostly instance mappings.
Let's break open one factory class, let's take:AttributeSelectorFactory
Observation:
It has a dictionary that stores CombinatorSymbol, SimpleSelector Attribute as key value pair.
Eg: Combinator Symbol for exactly is public static readonly String Exactly = "=";
Corresponding value is SimpleSelector AttrMatch

        public static SimpleSelector AttrMatch(String match, String value, String prefix = null)
        {
            var front = match;

            if (!String.IsNullOrEmpty(prefix))
            {
                front = FormFront(prefix, match);
                match = FormMatch(prefix, match);
            }

            var code = FormCode(front, "=", value.CssString());
            return new SimpleSelector(_ => _.GetAttribute(match).Is(value), Priority.OneClass, code);
        }

It is kinda, operator and their corresponding code.


2. HtmlParserOptions: This is a simple struct having four boolean fields saving properties like if the document IsEmbedded, IsScripting, IsScrictMode, callback on OnCreated
3. IBrowsingContext: A simple and lightweight browsing context having different EventListeners added on actions like Parsed, Requested, Requesting etc and having data regarding creator, history, security etc. Won't be going deep in here.

Cutting down, it in essence readys up the HtmlDocumetn for us that has basic information regarding the BrowsingContext, HtmlParserOptions and it has a proper mapping of what kind of operations to be performed in different supported formats.

parser.Parse(sourceCode);

            var document = CreateDocument(source);

            var parser = new HtmlDomBuilder(document);

            return parser.Parse(_options);

1. CreateDocument:

This does two things: 

(a) var textSource = new TextSource(source);

(b) var document = new HtmlDocument(_context, textSource);

textSource: It's just a stream abstraction to handle encoding and more.
HtmlDocument: Represents a document node that contains only HTML nodes.
It has methods like Clone(), LoadAsync, Get/Set title etc.

2. HtmlDomBuilder: It intakes this HtmlDocument, that is seemingly not yet parsed. HtmlDomBuilder essentially constructs the tree (as per described here: http://www.w3.org/html/wg/drafts/html/master/syntax.html ). It parses tokens, decides what is it, open tag, closed tag, formatting element, plain text, script etc, it wold be another blog post that we will surely do, in order to learn how is it doing this parsing, with a perspective on code architecture.

ParseAsync will prefetch tokens taking size and current position into consideration and making it an async job where as parse will simply take the entire document and will be running in the foreground as expected. But both are trying to create the DOM tree i.e. filling in the HtmlDocument "elements".
It will be interesting to walkthrough the Home and Foreign methods , code has taken care of what if foreign elements are present or the tag is not closed etc.
The connection between IElement and INode is intiguing. Please note
- The Element interface represents an object within a DOM document.
- The Node interface is an interface from which a number of DOM types inherit, and allows these various types to be treated similarly.

And Element implements Node interface, so it's bound to have the expected fields for ChildNodes , etc.
There are things like shadow roots and pseudo elements that will be needing attention in the next post too.

This is just a quarter walk into the code, I would be glad to cover the finer details in the upcoming post.

Hope it's helpful for someone trying to dive deep into the Angle Shrp code.
If you have queries, feel free to ping in comments or via mail, woul dbe happy to discuss.

See you the next post!
Soon enough !
Till then, keep hacking~

927d206cf9305e6c66b4088d947a006ff8ad9ed636b7cc690e

Caching

 Meanwhile, my cache went blank. No usage. How frequent are the chances of cache being blank if no use of a particular app is done?
*me makes a search engine query*

Hmm, so it has been rather a block on me restarting on the tech blogs I used to write. Truth is I wrote many, but couldn't finish them, so in draft. Well, this would be the feeble-most realization of writer's block if any.

Anyway, what am I intending to write here or will you just end up listening to a person who sounds tech and blabbers? Umm, no.
(Not for this post, for the least.)

So, here are quite a few realizations regarding tech I have been having lately.

One is tech at use. I don't know what software engineering has done to me but I end up finding a tech solution to my day-to-day problems.

One be "No-bai".
No, it's not a foreign language phrase though may seem like.
Just wanna say "no-bai" as in no maid, this problem happens when maid refuses to show up to work and you are unable to keep track of her because, well, mornings are meant for sleeping still. So maid steps in or steps out, you are mostly clueless until your sleeping ears turn sensitive to the adamant sounds of utensils she turns, overturns and places at some subtle surface so that it makes the loudest sound possible, thereby making it evident "She is working".

So, to solve this problem, I got a simple solution what if I ask her to just click a button on an app when she enters kitchen. What will that button do? Clicks her image and send it to server with timestamp :P
And then a server side small program with a rather simple interface to calculate number of days she worked.
The only issue was the app was supposed to be android and this can't be executed until my flatmate gets her new handset :P Even that is risky. Searched for some raspberry pi/arduino kinda thing but that will be more complicated to handle. I don't want her to leave the job anyway :P

Haah,
Next thing , umm, is the saddening impacts of big data. Be the consequences come up as some unexpected political victory or anything else, this is a true sadness prevalent all around. Humans being taken advantage of for having a psyche type, a personality, an emotion as it gets recorded in terms of likes or upvotes or shares or comments.
Just think how you began today? What all you did , what all apps you used and now just imagine the traces of your day and preferences you have left on web today!
Be it booking Uber or withdrawing 1500 INR or taking a selfie or ordering food or making a purchase online/offline via debit/credit cards. It's all recorded. Infact, just once for knowledge sake go to Google Settings and find out the data it collects. It even tracks which app you clicked how many times on your android device and in what order.
You think this is useless? Let's all just way to see how these patterns are being monetized over.
This is high time and calls action from all of us, atleast until we have a just law to govern us all, only awareness can help us.
I really appreciate the way Mozilla is driving the Data Privacy concerns. It's initiatives are in high regard.
Have a look here: https://blog.mozilla.org/blog/2017/01/26/data-privacy-day/

Only we can make a change here.
Just think twice before installing an Android app. Check for what all permissions it asks for.
Data theft from text messages is not a new thing. Elaborating a bit, suppose you gives some app access to read your text messages, now that app can read your text message inbox. Your inbox that has all your crucial details alongwith the bank updates. Would you like to let your bank details available to a third party app so easily?
Think twice.
Thing is be it apps or web services, they are not all free. Your data is the  new currency.
Stay conscious, stay aware!

PS: The post might be inconsistent or random, reason:external influences at the time of jotting down :P But this for sure breaks the pause on this blog. See you in the posts to be rolled ahead, in a rather much frequent manner, till then keep exploring and keep hacking.

Adieu!

Automate your facebook chats [JUST FOR FUN]

Hola folks!
 We all are on Facebook, yeah, and many times it happens you just don't wish to continue a conversation, reasons can be any and many. Say, you are not in a mood to talk, or though its a good friend who pinged, you are engrossed in some work in the parallel tabs and fb was just left open and hidden amongst the enormity of other tabs or may be its just another unwanted "hii .." (or maybe you are simply excited to automate the chat messages rather than responding in person).

Here is a way to deal with all those in a humane and polite way :)

Preferred browser: Chrome

Do this :
1. In the chatbox settings, click on "See Full Conversation"
2. Press F12 and write the code in console, make sure to make a necessary change before hitting enter, that is,  changing your reply button ID, since for every pageload on FB, this seemingly changes. Usually it is u_0_x though, but do inspect element on Reply Button and copy the id of the button and properly put it in the code.
3. Press Enter and let the fun begin.

var dialogs  = ["hey! ssup?", "oh okay", "nothing much at my end", "btw check out this standup comedy by Biswa https://www.youtube.com/watch?v=wkLmfIxi3No " , "kk", "ahaan", "dude, wait, I will brb", "gotta go, catch you later, bbye :) "];
    var chatBox  = document.getElementsByName("message_body")[0];  
    var replyBut = document.getElementById("u_jsonp_3_6");   //change this id , this may not match
    var diaNum=0;

    var chitInt   = setInterval(function() {
        chatBox.value = dialogs[diaNum] ;
     
        setTimeout(function(){
            replyBut.click();
        }, 4000);
        diaNum=diaNum+1;
      if (diaNum == 8) {
    clearInterval(chitInt);
  }
    }, 60000);

Feel free to play around with the code on my github repo here
Essentially this code is trying to put the dialogs in the array one by one in the chat box. So it starts with "hey!! ssup?" and end by "gotta go, catch you later, bbye :) ". Change these dialogs according to your chatting style to make it more realistic. Close the webpage to stop the script in between.
Also, for now it is one dialog every minute, change depending on the typing speed and availability of your friend (before starting the script).
Don't make it too less otherwise FB will pop up security check to assure its not a robot.

Personally, I got some really unexpected replies on these chats, like "Is this Aditi?" or "what's wrong with your texting language be? ". I needed to type in at times but it was fun overall.
Warning: People will leave frustrated or will be annoyed at times , so , sometimes you need to carry on a real chat just to make them normal and happy again.

I thank to all my friends, for tolerating the irrelevant messages while I was testing the script. Though some of them still don't know it was a script that conversed with them ( O yea B| ) .I am sorry.


PS: 1. Try at your own risk. In case FB blocks you or something crashes, its your responsibility.
2. Its meant for purely educational purposes :D

See you soon in the next post!

Happy Hacking!

Adieu.

Git Chit #1

Hola  Amigos!!

Diwali Greetings ! :)

So, this Diwali I had a few explosions with GIT,a tool budding developers hate as much as the pro developers adore.

[ LOL just check out the meaning of git :P It says "unpleasant or contemptible person". This will sooth you out if you have stumbled upon this post in frustration,desperately searching for your git solution. ]

So for those who are new to GIT, I must say it is a version control system, more here
In layman language, we all know is a complicated process to write and maintain code of projects especially when large number of people are involved, so git helps a lot. You can create branches, work in parallel on different features, undo the changes and do several other tasks involved in a simplified manner.

In this post we would be discussing a some use cases, i.e. what to do when a particular thing happens (based on my past experiences).

USE CASE #0
Usually when you are working on forked repo, you wish to get the changes done on the original repo too.You can easily do this by adding upstream.
git remote add upstream <git link of repo>

Now before doing any changes on local branch, do this:

1. git fetch upstream

2. git checkout master

3. git merge upstream/master

USE CASE #1

Deleting a file: (Do not do Shft+Del or Del directory. This will create problems.)

git rm <filepath>

git commit -m "Commit Message"

USE CASE #2
When you have done several not so useful commits, it becomes hard to follow in a PR, you can use git rebase. Eg for last five commits:  "git rebase -i HEAD~5" and squash them together and than git push --force ...
Have a look : https://www.atlassian.com/git/tutorials/rewriting-history/git-rebase-i

USE CASE #3
When working on different issues or features it is good to work on separate branches and create a separate PR from each one of them. This can be easily done by creating a new branch by using git branch -b <branch name>and checkout that, now work on that.
If you initially had just one branch and now wish to create a Pull Request with selected commits, use git cherry-pick feature, it's awesome!
So basically, do git log, it will show you all the commits. Now select the hash value of commit from which you which to create a new branch, so do :

You can create the branch via hash,

git branch branchname <sha1-of-commit>

or by using a symbolic ref.

git branch branchname HEAD~3

Now checkout to this branch,to include specific commits in it, use:
git cherry-pick <sha1-of-commit>

Do git log to get an idea of commits included in this branch.
Now do git push origin <branchname>
This will create the required PR.

So this was the first git chit.

Hope that helps!

Will see you in next post, hopefully soon, till then, keep hacking! :)

What emotions run through your playlist?

Hola !!

Long time. So I was upto creating this application one night , I named: "Playlist Emotions" , "What does the song says" :P , "PlayWithEmotions" and what not, each project of a failed experimentation.

Duh.

The project took way too long. Thanks to the noble idea of making and deploying it as a GWT application and then to improve upon the GUI of the app.
FYI , I tried but done neither of the things above. My Google Console developers trial account  supposedly has some issues with this app, yet to be resolved. So before the entire idea behind the app and the excitement of results it displays fades out , I thought, lemme write a blog post on the same.


So what is this all about?

The idea originated from a candid discussion with my friend on how songs influence our moods and also how our emotions affect the type of songs we listen to.

Being fascinated about knowing what kinds of songs I listen to, I thought of creating an application where I would just enter the folder location where my songs are and it would return me emotional composition of the songs I listen to.

Next stage was how to do that?

Logic is somewhat like this: When you share your music directory with me, I read the titles of all ,mp3 / .wav songs. Then the application extracts lyrics for those songs.

There exists no robust API that can give the lyrics of the song. Trouble multiplies when it is a Hindi Song, because we need to fetch the lyrics in English to do the sentiment analysis of the song.

So, indeed the program does a google search of "English Lyrics" + title of the song in your directory.
This gives me a list of links. I pick up the links resembling to either one of the following four trusted websites on lyrics , whichever is encountered first:
www.metrolyrics.com
www.lyricsfreak.com
www.hindilyrics.net (for Bollywood Songs)
www.bollymeaning.com (for Bollywood Songs)


Next step is scraping lyrics from these links. I studies the html structure of the websites and using JSoup library I finally fetched the string of my interest.

Then comes the third step of doing sentiment analysis, I did it using Synesketch .

"Synesketch analyses the emotional content of text sentences in terms of emotional types (happiness, sadness, anger, fear, disgust, and surprise), weights (how intense the emotion is), and a valence (is it positive or negative). The recognition technique is grounded on a refined keyword spotting method which employs a set of heuristic rules, a WordNet-based word lexicon, and a lexicon of emoticons and common abbreviations. Synesketch visualizes the emotions recognized in the form of real-time generative art. The art is partially based on Jared Tarbel’s algorithms and is inspired by the physics graphics of colliding particles."


And tadaa, I have the composition of emotions in songs I frequently listen too.
Mine was overall positive, with more of happy and surprise content :D If you want to try, find the app here:
https://github.com/nextLane/Playlist-Emotions

The code is open source, feel free to download and experiment. :)

This is how the raw application looks:

And when you get your results :D

The accuracy is challengable though, majorly because of two reasons:
1. Not the entire playlist is analysed, as in the trick that we applied to fetch the lyrics is usually applicable to around 70-80% songs, at times all, if you have named your songs properly in your playlist rather than random strings/numbers.

2. The Synesketch does sentiment analysis on words the song uses, we are equating the category of words with the category of songs, as in sad words corresponds to sad songs, This might not be the case always, for song also constitutes of musical beats , eh. Ok, just for clarification there is nothing like sad words, I just used it to indicate the set of words that can be categorized to a sad emotion. The details of it lie in the logic of Synesketch itself.
Detecting emotions in text is an ongoing research so high accuracy cannot be expected as of now.

So that was it, a whole night and the abstract thought of yesterday concreted into the application of today. Its fun to get your hands dirty into code this way.

See you in next post soon, with one more exciting creation/ exploration.

It would be great to hear your comments below!

Till then, keep hacking ;)

Adieu

Get back your original pendrive size

At times situations come when you have to get things back, just take this case.

You master at crashing your OS again and again, due to one reason or the other, and then starts the endless quest for free pen-drives and isos :P

The people who have had an experience of giving their noble laptops and pendrives to you usually are not that enthusiastic to help you again
(I don't know if their last experience was bad or what) but anyways , you are now in search of one pen-drive, a pen-drive that can bring you your life back.

Now lets say a heavily loaded 16 GB pen-drive somehow becomes available by someone you know, but the pen-drive is indeed of a person, the some one you know , knows (heavy responsibility).

You take  a look at the pen-drive, your intentions are not at all noble, you want to convert it into a bootable pen-drive that can bring the electric spark back to your laptop . So you silently copy all the contents to your harddrive, download Power ISO (on someone else's laptop :P) and make the PD bootable, by brutally clicking the format option.

So now you have the pd in your hand, time to get your OS back, and you eventually get it back.
Now, lets clean the footprints , eh... get back the data to pd, delete power ISO and return people's stuff back to them.

So you insert the PD again, but whats that? Unable to copy back the contents because the volume is squeezed to 2 MB !!

Random options and cost of the PD and your trust and reliability as a person is at stake suddenly.

Chuck the rest of the story, here is how to get it back:

The problem is that the remaining space of your USB flash disk has become unallocated. To repair it, use diskpart (in windows):

• Open command line (cmd) and type diskpart. Allow the windows to run it.
• type list disk, and find the number of your USB flash disk.
• type select disk X, and type the number of the USB disk instead of X.It should say that disk X is now selected.
• type clean.
• type create partition primary.

Now you should be able to use the flash disk as before. (You have to format it first)

Thanks to AskUbuntu.

So, now , return the stuff dude, and keep experimenting :P

Keep hacking!
See you in next post!

Continuing with SignalR

Hola amigo!!!

Ella unnav? (How are you? in Telugu :D )

Anyways, so here we continue with the second part of making apps with SignalR, a pretty cool library, for first part refer:

http://techvista-360.blogspot.in/2015/05/my-experiments-with-truth-oops-sorry.html

So struggling to figure out how to implement signalR in console apps that with .NET Client, I finally got the way through it.

You need to have two separate projects(preferably):

1. Server 

2. Client

Inside Server:

I have a main class, a startup class and a hub.

Before starting make sure you have added relevant references beforehand.

How?

Right click the project name and select Manage NuGet Packages, install:

signalR .NET client : Microsoft ASP.NET SignalR .NET Client

and  also SelfHosting 

One thing that comes into scens and is the most important is Cross Domain Communication.

Both the server and the client are separate projects now, so the client must be able to identify and connect to the server running in parallel as a separate project. What changes do we need to incorporate with the same?

Here they are:

Majorly the Startup.cs OWIN class gets changed to support cross domain connection.
Refer this for more details on what it is:

http://www.asp.net/signalr/overview/guide-to-the-api/hubs-api-guide-javascript-client#crossdomain

So the startup file is changed to:

using System;
using System.Threading.Tasks;
using Microsoft.Owin;
using Owin;
using Microsoft.Owin.Cors;
using Microsoft.AspNet.SignalR;
 
[assembly: OwinStartup(typeof(BroadcasterConsole.Startup))]
 
namespace BroadcasterConsole
{
    public class Startup
    {
        public void Configuration(IAppBuilder app)
        {
            app.Map("/signalr", map =>
            {
                map.UseCors(CorsOptions.AllowAll);
                var hubConfiguration = new HubConfiguration
                {
                    EnableJSONP = true
                    
                };
                map.RunSignalR(hubConfiguration);
            });
            
        }
    }
}

CorsOptions.AllowAll  will help your cross domain client to connect to te serevr. Also, please don't miss out the line:  [assembly: OwinStartup(typeof(BroadcasterConsole.Startup))]

Theses changes will make your server been detected and connected by the cross domain clients.

Rest of the server code is more or lesss the same as teh previous code. We need to have a hub class and a main class, I preferred to have them both in same namespace and it goes like this:

namespace BroadcasterConsole
{
    class Broadcast
    {
        static void Main(string[] args)
        {
            string url = "http://localhost:8088";
            try
            {
                using (WebApp.Start(url))
                {
                  //Have your code here
                }
 
        }
    }
 
 
 
 
    [HubName("broadcaster")]  
 
//Don't forget this line above, this name helps you to connect to the hub from the client side
    public class Broadcaster : Hub
    {
 
        public override Task OnConnected()
        {
            Console.WriteLine("Hub OnConnected {0}\n", Context.ConnectionId);
 
            return (base.OnConnected());
        }
 //Define your own hub methods
 
        public override Task OnReconnected()
        {
            Console.WriteLine("Hub OnReconnected {0}\n", Context.ConnectionId);
            return (base.OnDisconnected(true));
        }
    }
}



So, let's come back to the client side. I tried to have clients in different languages.

A peek into Javascript Client:

Firstly, include these scripts in this particular order:


   <!--Reference the jQuery library. -->
   <script src="Scripts/jquery-2.1.4.js"></script>
   <!--Reference the SignalR library. -->
   <script src="Scripts/jquery.signalR-2.2.0.js"></script>
 
 
   <!--Reference the autogenerated SignalR hub script. -->
 
   <script src="http://localhost:8088/signalr/hubs" type="text/javascript"></script>

Please see I have hosted my server on 8088 port, change it accordingly, as per the url of your hosted server.

Here goes your main connection code:

<script type="text/javascript">
       
       $.connection.hub.url = 'http://localhost:8088/signalr';
       var hubProxy = $.connection.broadcaster;
       hubProxy.client.addMessage = function (message) {
     
           //message is a c# struct Request object.
    
           // Add the message to the page.
           $('#discussion').append('<li><strong> ' + document.getElementById('displayname').value + ' received request  Name:' + message.Name +  '\n   Program:' + message.Program +  '\n     Input:' + message.Input
               + '</strong>:&nbsp;&nbsp;</li>');
       };
 
       $.connection.hub.start().done(function(){
       
           alert('Now connected, connection ID=' + $.connection.hub.id);
           $('#displayname').val(prompt('Which process are you?', ''));
           hubProxy.server.registerMe(document.getElementById('displayname').value);
           $('#sendoutput').click(function () {
 
               hubProxy.server.getClientOutput({ "Name": "JS Client", "Result": $('#output').val() });
 
               $('#output').val('').focus();
           });
       });
     
      
   </script>

Its fine if you don't get any of the above code since it would be a bit out of the context you are into. just focus on highlighted lines.

Set the hub url, create a hub Proxy.

Prior to starting teh connection, add the client side methods that the hub can call, for example here we have: addMessage.

Then start the connection by this line:  $.connection.hub.start().done.

You can call server side methods by : hubProxy.server.getClientOutput (//required params)

There we finish with the JavaScript client, we can do the same with clients for different languages:


.NET Client:


//Set connection
           var connection = new HubConnection("http://localhost:8088/");
//Make proxy to hub based on hub name on server
           myHub = connection.CreateHubProxy("Broadcaster");
           
//Add client side methods

           myHub.On<Request>("addMessage", param =>{
              //Define your method
           });
 
//Start connection
           connection.Start().ContinueWith(task =>
           {
               if (task.IsFaulted)
               {
                   Console.WriteLine("There was an error opening the connection:{0}",
                                     task.Exception.GetBaseException());
               }
               else
               {
                   Console.WriteLine("Connected");
 
               }
 
           }).Wait();
 //Invoking hub methods

           myHub.Invoke<string>(<Method Name as is>, clientName)


C++ Client:
Here is a detailed guide to do the same with C++. 
https://github.com/aspnet/SignalR-Client-Cpp








It is pretty simple to use, in case you have any queries, leave in comments below.
So that was all folks, see you in next post!
Till then, 
Happy Hacking!
Meanwhile, get your signals across using SignalR :D