This is part 3 of a series on KiGG Automation. Code in this article may be dependent on previous entries.

Part 1: Automating KiGG Publishing
Part 2: KiGG’s Story Summary 
Part 3: Automating KiGG Submission

When you’re first starting a KiGG site and your user base is low, creating a bot can ease the pain of obtaining content. However, in the default install of KiGG there is no way to automate this. You can search blogs and news sites, but submitting articles from those sites is a manual process; even for the sources you trust.

My goal in this article is to build off of the first article so that submission can be automated. There are many sources of content out there, and I may cover some aspects of retrieving this in future articles. But for now, that it is up to you to explore.

Finding the submit route is pretty simple: it is Submit and it is handled by the StoryController.

_routes.MapRoute("Submit", "Submit", 
    new { controller = "Story", action = "Submit" });

The method on the controller makes it clear what http method we must use and the parameters required.

[AcceptVerbs(HttpVerbs.Post), ValidateInput(false), Compress]
public ActionResult Submit(string url, 
    string title, string category, string description, string tags)

Analyzing this method shows that another JSON class can be used: JsonCreateViewData. We will need to add this to our bot’s codebase. It uses two properties already found on KiggResult, so we will call this KiggCreateResult and derive from the former class.

public class KiggCreateResult : KiggResult
{        
    [JsonProperty(PropertyName = "url")]
    public string Url { get; set; }

    public static KiggCreateResult Create(string json)
    {
        return JsonConvert.DeserializeObject<KiggCreateResult>(json);
    }
}

Now, inside KiggBot.cs, we will need to add our submit method. I think it may be better to return the url to the story here, but I’ll worry about refactoring this later. Remember that HttpDictionary will handle the form encoding for us.

public bool Submit(string storyUrl, string title, 
    string category, string description, string tags)
{
    HttpDictionary dictionary = new HttpDictionary { 
        {"url", storyUrl}, 
        {"title", title},
        {"category", category},
        {"description", description},
        {"tags", tags}
    };

    var result = KiggCreateResult.Create(
        session.Post(url + "Submit", dictionary));

    return result.IsSuccessful;
}

That seemed easy enough. Now, in the test program I will call this with a few parameters to see how it works.

KiggBot bot = new KiggBot("http://localhost:1736");
bot.Login("bee", "iambusy");
bot.Submit("http://example.com/", "Example", "Agile", 
    "This is a test", "Example, News");
bot.Logout();

No dice. I received an error that the user agent must be set. I think user agent should be configurable in our program, but for simplicity I will add a constant to WebSession for userAgent and set it on the request object in the Post method.

private const string userAgent = "Pigg";
public string Post(string url, string formData)
{
    var request = WebRequest.Create(url) as HttpWebRequest;
    request.UserAgent = userAgent;
    ...
}

I tried again, and it worked perfectly. The bot created an entry with two new tags: Example and News. Keep in mind that Category must match a valid category in your KiGG install.

If you’re following along, you can now automate KiGG publishing and article submission. You can also define what part of the html from which the story summary is retrieved. Leave a comment on what part of KiGG automation you would like me to look at next.

This is part 2 of a series on KiGG Automation.

Part 1: Automating KiGG Publishing
Part 2: KiGG’s Story Summary 
Part 3: Automating KiGG Submission

I noticed that sometimes when you submit a story to KiGG, the summary of the story looks rather garbled. Wishing to correct this, I explored what is actually happening when KiGG summarizes a url for you.

The first step in the process is to go to the submit new story page. When you enter a url into the url field, the site will perform an ajax call that suggests a title and summary. This call is defined by _urlChanged() in story.js. The ajax call is to /Retrieve. Here is the relevant code from New.aspx.

if (Model.AutoDiscover)
{
    scriptManager.RegisterOnReady("Story.set_retrieveStoryUrl('{0}');"
        .FormatWith(Url.RouteUrl("Retrieve")));
}

This will only work if AutoDiscover is set to true, and that is defined in the web.config.

After looking at our routes, we can tell that /Retrieve is calling the Retrieve method on the StoryController. It then returns a view with a StoryContent object after calling the Get method on the member _contentService. This member is set in the constructor. Since KiGG uses Unity, we will need to follow the trail in the web.config to determine which IContentService it’s using (or you can view all classes that implement IContentService and infer the correct choice).

Looking in the web.config, we find the tag: <type type=”StoryController”>. Within the constructor element, we find that contentService is defined as IContentService. Looking through the type elements, we discover that there are three IContentService types. Two of these are decorators (Logging and Caching), so the functionality we’re looking for must be in the one named Base. It maps to ContentService, which is defined in the typeAlias section as Kigg.Infrastructure.ContentService, Kigg.Core. The cool thing about Unity is that you can add your own ContentService decorators and use the web.config to decorate the class without changing any source code.

The content service retrieves the html from the url provided, then calls the converter (again defined by Unity).

string html = 
    _httpForm.Get(new HttpFormGetRequest{ Url = url }).Response;

return string.IsNullOrEmpty(html) ? 
    StoryContent.Empty : _converter.Convert(url, html);

The converter maps to HtmlToStoryContentConverter. Inside that class is the code that attempts to find the content.

private Node TryToFindContentNode(Node bodyNode)
{
    Node contentNode = null;

    foreach (string xPath in _xPaths)
    {
        contentNode = bodyNode.SelectSingleNode(xPath);

        if (contentNode != null)
        {
            break;
        }
    }

    return contentNode;
}

_xPaths is passed in a constructor which is called by another constructor which reads a file. Inside the web.config, we find that Unity has defined the location of the file.

<param name="fileName" parameterType="System.String">
    <value type="System.String" value="App_Data/contentNodes.txt"/>
</param>

Our content nodes are defined within App_Data/contentNodes.txt file. Using this file, we can add xpath style declarations to properly retrieve content from nonstandard articles. Further logic (ignoring h1s, etc) will require changes to the ContentService itself.

This is part 1 of a series on KiGG Automation.

Part 1: Automating KiGG Publishing
Part 2: KiGG’s Story Summary 
Part 3: Automating KiGG Submission

When I set up my first KiGG site, I was surprised to discover that I had to manually publish articles. I assumed it would be an automated process that would run once a day. Since there are times I may not be able to log into my website, I set about figuring out how to automate the process.

I should note that during this process, I didn’t use best practices. I had one requirement: make a program that I can schedule to publish stories on KiGG. I wasn’t really sure what I would need to go about doing that. So, I did what I assume most developers do: create a console application prototype. I don’t claim this code is perfect, but if you want classes to automate publishing in KiGG (or control MVC applications), this will do the trick.

KiGG is an ASP.NET MVC application. This made it easy to figure out where to begin. I wanted to Publish, so I needed to find the admin link and see what it called. I guessed it would be something like, /Publish, and I was correct.

scriptManager.RegisterOnReady("Administration.set_publishUrl('{0}');"
    .FormatWith(Url.RouteUrl("Publish")));

set_publishUrl is simply a setter in the JavaScript to prevent a url from being hardcoded in script. Looking over the JavaScript I could tell there wasn’t much more needed than calling the proper url.

$.ajax(
    {
        url: Administration._publishUrl,
        type: 'POST',
        dataType: 'json',
        data : '__MVCASYNCPOST=true', // a fake param to fool iis for content-lenth,
        beforeSend: function()
        {
            $U.showProgress('Publishing stories...');
        },
        success: function(result)
        {
            $U.hideProgress();

            if (result.isSuccessful)
            {
                $U.messageBox('Success', 'Story publishing process completed.', false);
            }
            else
            {
                $U.messageBox('Error', result.errorMessage, true);
            }
        }
    }
);

The script is doing a POST on the /Publish url with fake data (to fool IIS?).

Next, it was time to check the routing for /Publish. Routing in KiGG is defined in Kigg.Web\BootstrapperTasks\RegisterRoutes.cs. It was done this way to be consistent with the Open Closed Principle.

_routes.MapRoute("Publish", "Publish", 
    new { controller = "Story", action = "Publish" });

Now we know the entry to this is in the story controller and that the method is Publish. The story controller being used is defined within the web.config file using Unity.

<typeAlias alias="StoryController" type="Kigg.Web.StoryController, Kigg.Web"/>

I set up a break point and wrote a quick routine to post to the publish url.

var request = WebRequest.Create(http://localhost:1736/Publish);
request.ContentType = "application/x-www-form-urlencoded";
request.Method = "POST";

var form = Encoding.ASCII.GetBytes("__MVCASYNCPOST=true");
using (Stream requestStream = request.GetRequestStream())
{
    requestStream.Write(form, 0, form.Length);
    requestStream.Close();
}

var response = request.GetResponse();
using (StreamReader reader = new StreamReader(response.GetResponseStream()))
{
    var responseText = reader.ReadToEnd();
    Debug.WriteLine(responseText);
    reader.Close();
}

Everything executed fine, but the return message indicated something was wrong: “{"isSuccessful":false,"errorMessage":"You are currently not authenticated."}.” It’s pretty obvious that authentication would be necessary, otherwise I could publish DotNetShoutOut whenever I wanted. Unfortunately, the obvious way to authenticate does not work.

request.Credentials = new NetworkCredential("admin", "password");

Since it is a NetworkCredential, I assume this code would work with Windows authentication turned on. However, we’re using forms authentication. I looked at the routes and found one for Login and Logout. At this point it was clear that I would be calling post multiple times, so I moved the code to a method that would accept a url and formdata as a string.

Post("http://localhost:1736/Login", "userName=admin&password=password");
Post("http://localhost:1736/Publish", "__MVCASYNCPOST=true");
Post("http://localhost:1736/Logout", "__MVCASYNCPOST=true");

This returned the following results:

{"isSuccessful":true,"errorMessage":null}
{"isSuccessful":false,"errorMessage":"You are currently not authenticated."}
{"isSuccessful":false,"errorMessage":"You are currently not logged in."}

I put a breakpoint in the Login method of the MembershipController, and from there was able to see what was happening. Although it was a few levels deep, it was clear that FormsAuthentication.SetAuthCookie was the important piece of code; I needed a way to retain cookies between requests.

Cookies are only available by using HttpWebRequest and HttpWebReponse rather than their base classes. So, I cast my variables to the proper class. One odd thing I found was the cookies have to be retrieved from the request rather than the response after you have requested the response. I wrapped this functionality up in a WebSession class.

public class WebSession
{
    protected CookieCollection Cookies { get; set; }

    public WebSession()
    {
        Cookies = new CookieCollection();
    }

    public string Post(string url, HttpDictionary formData)
    {
        return Post(url, formData.ToString());
    }

    public string Post(string url, string formData)
    {            
        var request = WebRequest.Create(url) as HttpWebRequest;
        if (request == null)
        {
            throw new HttpException();
        }
        request.ContentType = "application/x-www-form-urlencoded";
        request.Method = "POST";
        request.CookieContainer = new CookieContainer();
        request.CookieContainer.Add(request.RequestUri, Cookies);

        var form = Encoding.ASCII.GetBytes(formData);
        WriteToRequest(request, form);

        var response = request.GetResponse() as HttpWebResponse;
        if (response == null)
        {
            throw new HttpException();
        }

        var responseText = ReadResponse(response);
        Cookies = request.CookieContainer.GetCookies(request.RequestUri);

        Debug.WriteLine(responseText);
        
        return responseText;
    }

    private static void WriteToRequest(WebRequest request, byte[] form)
    {
        using (Stream requestStream = request.GetRequestStream())
        {
            requestStream.Write(form, 0, form.Length);
            requestStream.Close();
        }
    }

    private string ReadResponse(WebResponse response)
    {
        using (StreamReader reader = 
            new StreamReader(response.GetResponseStream()))
        {
            var responseText = reader.ReadToEnd();
            reader.Close();
            return responseText;
        }
    }
}

If you look closely you’ll notice an HttpDictionary class. This is because I wanted to treat the form data as a dictionary rather than a string. I suppose this could be written as an extension method on Dictionary<string, string> instead.

public class HttpDictionary : Dictionary<string, string>
{
    public override string ToString()
    {
        return this.Select(q => EncodePair(q))
            .Aggregate((a, b) => a + "&" + b);
    }

    private string EncodePair(KeyValuePair<string, string> pair)
    {
        return HttpUtility.HtmlEncode(pair.Key) +
            "=" + HttpUtility.HtmlEncode(pair.Value);
    }
}

I also felt it necessary to wrap up the result received from KiGG in a class that can be used. Otherwise, it’s difficult to know whether or not the call was successful. To convert from Json, I used Json.NET.

public class KiggResult
{        
    [JsonProperty(PropertyName="isSuccessful")]
    public bool IsSuccessful{ get; set; }

    [JsonProperty(PropertyName = "errorMessage")]
    public string ErrorMessage { get; set; }

    public static KiggResult Create(string json)
    {
        return JsonConvert.DeserializeObject<KiggResult>(json);
    }
}

Finally, I made a KiggBot class to provide us an interface that makes sense for accessing a KiGG site. This can be changed to throw an exception if the result indicates the call was not successful.

public class KiggBot 
{
    private static readonly HttpDictionary mvcAsyncPost = new HttpDictionary
    {
        {"__MVCASYNCPOST", "true"}
    };

    private WebSession session;
    private string url;

    public KiggBot(string url)
    {
        if (!url.EndsWith("/"))
            url += "/";

        this.url = url;
        session = new WebSession();
    }

    public bool Login(string userName, string password)
    {
        HttpDictionary dictionary = new HttpDictionary { 
            {"userName", userName}, {"password", password}};

        var result = KiggResult.Create(
            session.Post(url + "Login", dictionary));
        
        return result.IsSuccessful;
    }

    public bool Logout()
    {
        var result = KiggResult.Create(
            session.Post(url + "Logout", mvcAsyncPost));

        return result.IsSuccessful;
    }

    public bool Publish()
    {
        var result = KiggResult.Create(
            session.Post(url + "Publish", mvcAsyncPost));

        return result.IsSuccessful;
    }
}

With these classes, it’s fairly easy to create an automated publishing program. At the very least, write a console app and use scheduler to fire it daily.

KiggBot bot = new KiggBot("http://localhost:1736");
bot.Login("admin", "password");
bot.Publish();
bot.Logout();

I recently deployed a KiGG site, but I wanted it to act more like DZone than DotNetShoutOut. On DotNetShoutOut, if you click the link from any of non-detailed page views, it takes you to the site. DZone will take you to the detailed view of the story, and if you click the link from there it will take you story’s site. The reason for this change is that I want my site to foster the community: more comments and viewing promoters. Eventually, I would like to code up a friend system for KiGG similar to DotNetKicks friend system, and I feel that seeing who promoted a story will grow more connections between those in the community.

Here’s how you can implement viewing the detailed story. In your KiGG 2.2 release codebase, open up Kigg.Web\Views\Story\Story.ascx. On line 132, you will find the declaration for detailUrl.

<% string detailUrl = Url.RouteUrl("Detail", new { name = story.UniqueName }); %>

Cut this and paste it immediately after the table tag on line 99. Then, replace the first td’s content with the following snippet of code.

<% if (Model.DetailMode) %>
<% { %>
<h2><a class="entry-title taggedlink" rel="bookmark external" 
        href="<%= Html.AttributeEncode(story.Url) %>" target="_blank" 
        onclick="javascript:Story.click('<%= attributedEncodedStoryId %>')">
        <%= Html.Encode(story.Title)%></a></h2>
<% } %>
<% else %>
<% { %>
<h2><a class="entry-title taggedlink" 
        href="<%= Html.AttributeEncode(detailUrl) %>">
        <%= Html.Encode(story.Title)%></a></h2>
<% } %>

That’s it: simple change! I think it would be better to make this configurable and submit it to the project. I may do that when I have more time to sift through the codebase.