Well this was interesting experience! You would think this would be very simple to write a Twitter client. Right?! I was confronted with many interesting problems. Service was thrown at me all kinds of curve balls. Malformed XML, using protocol status codes to raise errors, different message shapes for a single operation. Come see how WCF architecture was handling it all through custom bindings, operation behavior and channel modifications.
How do you go about creating WCF client for Twitter service?
The first step is figuring out the end point communication method, security requirements and message shape of the service APIs. Twitter API is located here http://groups.google.com/group/twitter-development-talk/web/api-documentation
In the case of Twitter its API request model uses REST for service calls, HTTP Basic Auth for security model and response message format is either POX (Plain Old XML) or JSON. I chose to implement POX because there are plenty of tools in CLR base class library that can manipulate XML.
How does this translate to WCF client configuration?
We can configure WCF client many ways. One of my objectives is that WCF client implementation (aka proxy) should be the most basic vanilla C# implementation out of the box I can get away with.
This is what I wanted the client code to look like:
using (TwitterClient proxy = new TwitterClient("[USER]", "[PASSWORD]"))
{
var response = proxy.Update("hello");
Console.WriteLine(response);
}Accompanying this client is the configuration file with settings that will configure our proxy at runtime. The key element that will bind our proxy with this endpoint is the contract, ITwitterClient.
1: <client>
2: <endpoint
3: address="http://twitter.com"
4: binding="customBinding"
5: contract="ITwitterClient"
6:
7: behaviorConfiguration="WebGet"
8: bindingConfiguration="TwitterBinding"
9:
10: name="TwitterClient" />
11: </client>
The obvious ABCs of WCF: address - where is the service located; Binding - how is the WCF runtime configured to communicate with the service; Contract - what the service communicates.
Enabling REST
Looking at the config file will not give you any hints just yet about how did I configure proxy to use REST or Basic auth. To get the REST model I had to tell the end point to use different behavior than standard SOAP. You turn this on by modifying the end point behavior and adding webHttp behavior.
1: <behaviors>
2: <endpointBehaviors>
3: <behavior name="WebGet">
4: <webHttp />
5: </behavior>
6: </endpointBehaviors>
7: </behaviors>
This behavior modifier enables the Web programming model. What this means is that you can use POST, GET and other standard web methods and use RESTful service call pattern.
Configuring Basic Authentication
This could have been dead simple to configure but I had to deal with some unusual service behaviors and configure my endpoint to use custom binding. More about this latter in this article. In normal circumstances I could have just used new webHttpBinding that comes with WCF 3.5 and configure it like this:
<webHttpBinding>
<binding name="BasicAuth">
<security mode="TransportCredentialOnly" >
<transport clientCredentialType="Basic"/>
</security>
</binding>
</webHttpBinding>
But I could not use this binding. I had to create my own custom binding.
Client implementation
If you look at any proxy auto-generated code you will find that all proxy clients derive from ClientBase<T> class. I did not generate the client but I created my own following the same idea:
I wanted to keep with "out of the box" code as much as I can. You might notice that there are bunch of ITiwtterxxxx interface implementations.
Why this many interfaces?
If you look through the Twitter API you will notice that the API is divided in different method sections. It seemed logical to divide these into different interfaces for clarity and give opportunity for developers to implement clients that focus on different functions. Twitter client inherits from all these interfaces.
[ServiceContract]
public interface ITwitterClient : ITwitterStatus, ITwitterHelp, ITwitterBlock, ITwitterNotification,
ITwitterFavorite, ITwitterAccount, ITwitterFriendship, ITwitterDirectMessage, ITwitterUser
{
}Each of the interfaces is a service contract and defines operation contracts for each method in the API specification.
WebGet and WebInvoke combined with webHttp end point behavior enable Web http programing. WebGet operations should be logical retrieval operations while WebInvoke represents HTTP invocation methods i.e. POST, PUT, DELETE. Both of these attributes accept the UriTemplate.
For example, FriendsTimeline service operation from above requires user to be passed in. Template below shows how to configure URL to use the passed in variable.
Problems and solutions
Issuing first call to the service I was presented with immediate problem.
DataContract serializer and arrays
First obstacle was trying to figure out how to map the array of elements coming from the Twitter. Calling FriendsTimeline method returns this response:
<?xml version="1.0" encoding="UTF-8"?>
<statuses type="array">
<status>
<created_at>Sat Jun 21 01:27:49 +0000 2008</created_at>
<id>839961831</id>
<text>hello</text>
<source>web</source>
<truncated>false</truncated>
<in_reply_to_status_id></in_reply_to_status_id>
<in_reply_to_user_id></in_reply_to_user_id>
<favorited>false</favorited>
<user>
<id>15161276</id>
<name>[user name]</name>
<screen_name>[screen_name]</screen_name>
<location></location>
<description></description>
<profile_image_url>http://static.twitter.com/images/default_profile_normal.png</profile_image_url>
<url></url>
</user>
</status>
</statuses>I wanted to use DataContract serializer (keeping with out of the box spirit). This provided to be very easy to do. Just inherit from List<T>. ItemName refers to another type that serializer will use to create instances of the items contained in the list.
namespace Vertigo.WCF.TwitterClient.API
{
[CollectionDataContract(Name = "statuses", ItemName = "status", Namespace = "")]
public class Statuses : List<Status>, IResponseErrorProvider
{
public ResponseError ServiceError { get; set; }
}
}Here you also have a hint of how I am going to handle errors returned by the service.
Returning XML fragments
Calling an API method VerifyCredentials returns just this XML fragment:
<authorized>true</authorized>
I could not find a way to de-serialize this response into an instance using plain classes and DataContractSerializer. I resorted to implement IXmlSerializable and handle this problem my self (detailed exception checking not included, this is just a demo).
namespace Vertigo.WCF.TwitterClient.API.Helpers
{
[Serializable]
public class BooleanResponseWrapperBase : IXmlSerializable
{
public bool Test { get; set; }
#region IXmlSerializable Members
public System.Xml.Schema.XmlSchema GetSchema()
{
return null;
}
public void ReadXml(System.Xml.XmlReader reader)
{
reader.MoveToContent();
reader.Read();
Test = reader.ReadContentAsBoolean();
}
public void WriteXml(System.Xml.XmlWriter writer)
{
//noop
}
#endregion
}
}Now I can add a wrapper class for the authorized response like this:
[Serializable]
[XmlRoot("authorized")]
public class VerifyCredentialsResponseWrapper : BooleanResponseWrapperBase
{
}WCF has many extension points and I was considering using the MessageContract to solve this problem but MessageContract still required to have well formed XML coming from the service. It would have been better if service responded with proper type encapsulation:
<Verication>
<authorized>true</authorized>
</Verification>This could be then easily mapped to
public class Verification
{
public bool Authorized { get; set; }
}and DataContractSerializer would be happy.
Handling HTTP 403 status
Well this was an interesting wrinkle. When service encounters an error, for example you try to add non existing user as your friend, service will return 403 status code and send you XML payload with error. Using Microsoft Network Monitor here is the response from the server:
Frame:
Ethernet: Etype = Internet IP (IPv4)
Ipv4: Next Protocol = TCP, Packet ID = 701, Total IP Length = 929
Tcp: Flags=...PA..., SrcPort=HTTP(80), DstPort=60739, Len=889, Seq=1638460779 - 1638461668, Ack=441960162, Win=64838 (scale factor not found)
Http: Response, HTTP/1.1, Status Code = 403
- Response:
ProtocolVersion: HTTP/1.1
StatusCode: 403, Forbidden
Reason: Forbidden
Via: 1.1 twitter-web016.twitter.com, 1.1 ISA
Connection: Keep-Alive
Proxy-Connection: Keep-Alive
ContentLength: 169
Expires: Mon, 23 Jun 2008 21:07:10 GMT
Date: Mon, 23 Jun 2008 20:37:10 GMT
ContentType: application/xml; charset=utf-8
Server: hi
Status: 403 Forbidden
P3P: CP="NOI DSP COR NID ADMa OPTa OUR NOR"
X-Runtime: 0.00992
Cache-Control: no-cache, max-age=1800
Set-Cookie: lang=; path=/
Set-Cookie: _twitter_sess=BAh7CDoJdXNlcmkDvFfnOgdpZCIlNDg4OGMyM2Y2MjI4NDEyMzVlNGNiMTNi%250AYTUyNTQzN2YiCmZsYXNoSUM6J0FjdGlvbkNvbnRyb2xsZXI6OkZsYXNoOjpG%250AbGFzaEhhc2h7AAY6CkB1c2VkewA%253D--f8a906186ec107cbb92d6dfda23050b11aebff78; domain=.twitter.com; path=
Vary: Accept-Encoding
HeaderEnd: CRLF
- payload: HttpContentType = application/xml; charset=utf-8
- XmlPayload:
<?xml version="1.0" encoding="UTF-8"?>
- <hash>
- <error>
Could not follow user: User not found.
</error>
- <request>
/friendships/create/petarvucetin1.xml
</request>
</hash>
Of course this did not sit well with the WCF infrastructure. WCF was throwing an exception
System.ServiceModel.Security.MessageSecurityException was unhandled
Message="The HTTP request was forbidden with client authentication scheme 'Basic'."
Source="mscorlib"
StackTrace:
InnerException: System.Net.WebException
Message="The remote server returned an error: (403) Forbidden."
Which is correct response. So how do I fool the WCF and let me handle this error? WCF channel extensibility, of course!
WCF Channel, Operation and Message Customization
I need some way to intercept the exception on the transport level, handle it and then re-shape the message in such a way that it can be de-serialized by DCS (DataContractSerializer). This is where I started from Nicholas Allen - ReplyMangler Channel.
If you remember our client configuration looked something like this:
1: <client>
2: <endpoint
3: address="http://twitter.com"
4: binding="customBinding"
5: contract="ITwitterClient"
6:
7: behaviorConfiguration="WebGet"
8: bindingConfiguration="TwitterBinding"
9:
10: name="TwitterClient" />
11: </client>
We are using a custom binding specified by bindingConfiguration attribute (line #8).
<bindings>
<customBinding>
<binding name="TwitterBinding">
<webMessageEncoding />
<WebReqestExceptionsInterceptor />
<httpTransport manualAddressing="true" authenticationScheme="Basic" />
</binding>
</customBinding>
</bindings>Custom binding is actually real binding in WCF BCL. This binding allows you to configure your channel stack any way you want. The only requirement is that you have at least one transport protocol and message encoding protocol.
If you would to crack open WebHttpBinding with reflector you would see that it contains two binding elements
I have added new custom binding element WebRequestExceptionInterceptor and registered it with WCF inside the config file.
Callout A shows my custom element inserted between the message encoder and transport. Callout B shows how to register your own custom binding element extension.
The important thing to note is that order of binding elements does matter. There is a lot of plumbing code for putting this in place so here is the most important part.
Callout A shows the capture of the inner channel exception. When exception is thrown we need to create new response message. To the response message we need to add HttpResponse property. I also insert new property that someone else can use to test if we have "fixed" up the message (callout B).
Now that this is done something needs to handle this message on operation level. This is where we add MessageFixer operation behavior.
Modifying operation behavior
The final thing in our chain of changes is converting the message to something that DataContractSerializer can understand. The simplest way to do this is to modify the operation behavior.
/// <summary>
/// Befriends the user specified in the ID parameter as the authenticating user. Returns the befriended user in the requested format when successful. Returns a string describing the failure condition when unsuccessful.
/// </summary>
/// <param name="user"></param>
/// <returns></returns>
[OperationContract]
[MessageFixer]
[WebInvoke(UriTemplate = TwitterURITemplates.Friendship.Create, BodyStyle = WebMessageBodyStyle.WrappedRequest, RequestFormat = WebMessageFormat.Xml, ResponseFormat = WebMessageFormat.Xml)]
User AddFriend(string user);
I have created MessageFixer who's job is to inspect the message and fix it when we get an null response or 403 protocol exception. The way message fixer works is by overriding the message formatter for this operation. We hook into the operation behavior when ApplyClientBehvior is called.
namespace Vertigo.WCF.TwitterClient.Customizations
{
[AttributeUsage(AttributeTargets.Method)]
public class MessageFixer : Attribute, IOperationBehavior, IClientMessageFormatter
{
#region const
private const string NullMessage = "nil-classes";
private const string ErrorMessage = "hash";
private const string ResponseErrorKey = "Service403Response";
#endregion
#region fields
IClientMessageFormatter _formatter;
Type _return;
#endregion
#region IOperationBehavior Members
public void AddBindingParameters(OperationDescription operationDescription, BindingParameterCollection bindingParameters)
{
}
public void ApplyClientBehavior(OperationDescription operationDescription, ClientOperation clientOperation)
{
MessageDescription md = operationDescription.Messages.First( m => m.Direction == MessageDirection.Output);
_return = md.Body.ReturnValue.Type;
_formatter = clientOperation.Formatter;
clientOperation.Formatter = this;
}First we remember the original formatter and then insert MessageFixer as new client formatter.
#region IClientMessageFormatter Members
public object DeserializeReply(Message message, object[] parameters)
{
object helperInstance = Activator.CreateInstance(_return);
//we have special condition where service sets Http response code to 403 that signals that an error has occured
KeyValuePair<string,object> serviceErrorProperty = message.Properties.FirstOrDefault(p => p.Key == ResponseErrorKey);
if (serviceErrorProperty.Key != null)
{
//we have an error message
IResponseErrorProvider responseErrorProvider = helperInstance as IResponseErrorProvider;
if (responseErrorProvider != null)
{
//unpack the error payload from message and assign to the object
ResponseError payload = message.GetBody<ResponseError>();
responseErrorProvider.ServiceError = payload;
//return fixed null type with error attached to it
return helperInstance;
}
}
//another message we might get is <nil-classes type="array"/> for empty arrays.
XmlDictionaryReader xdr = message.GetReaderAtBodyContents();
xdr.MoveToContent();
if (xdr.Name == NullMessage)
{
return helperInstance; //standin for the null value
}
return _formatter.DeserializeReply(message, parameters);
}
public Message SerializeRequest(MessageVersion messageVersion, object[] parameters)
{
return _formatter.SerializeRequest(messageVersion, parameters);
}
#endregion
For request serialization we just use the original formatter. When we get request to deserialize the message this is where we need to do the message fixing. If message properties contain our custom service error property and the type supports error assignment via IResponseErrorProvider we can use message GetBody<T> to deserialize the payload to an instance type and assign this error instance to the return type.
I had an option to raise the exception from the channel when service reported an error. The reason I did not go with this route is that raising exception would fault the channel and it would not be usable. Another reason is that I am not sure how client should behave (at this time) when service raises an error.
In case of the null message, that is the message contains <nill-classes> element we just return helper instance of the type we need to de-serialize.
UriTemplate and Enum method parameters
When using URI templates there is no way to dictate how "serialization" of the parameter is performed when substituting it in URI template. For example update delivery device URI template accepts an enumeration of devices in its template:
public const string UpdateDeliveryDevice = "/account/update_delivery_device.xml?device={device}";Service is expecting the name of the device to be all lower case for example:
http://www.twitter.com/account/update_delivery_device.xml?device=sms
[OperationContract]
[WebInvoke(UriTemplate = TwitterURITemplates.Account.UpdateDeliveryDevice, BodyStyle = WebMessageBodyStyle.Bare, RequestFormat = WebMessageFormat.Xml, ResponseFormat = WebMessageFormat.Xml)]
void UpdateDeliveryDevice(Devices device);
When URI template is constructed, value of the parameter is extracted by calling the ToString() of the type. It would have been nice if there was a way to control this behavior either through XmlEnum or TypeConverter. Instead I had to create enum that matched exactly what service is expecting!
//There is no simple way to control how WCF uri template
// is serialized withouth super heavy lifting. The service uses lower case names and there is no
// easy way to convert from proper case to lower case when WCF is constructing the URL for the call.
public enum Devices
{
none,
im,
sms
}DebuggerDisplay
Do provide this very usefully attribute to your classes for easier debugging like so:
[DataContract(Name = "user", Namespace = "")]
[System.Diagnostics.DebuggerDisplay("Name = {Name}, ScreenName = {ScreenName}")]
public class User : IResponseErrorProvider
{
[DataMember(Name="id", Order = 0)]
public string ID { get; set; }Conclusion
I am in no position to criticize the Twitter API but I must say that API designers did not follow some basic rules when creating the public API. I whish that API designers pay more attention to client usability issues when creating new APIs. For example:
- Publish your API documentation near your web site, for example http://twitter.com/Api
- Keep your API documentation and examples up to date
- If you have decided to publish the API please make sure you provide working examples or at least detailed documentation on request and response formats
- Document VERY clearly what each operation input and output should look like with example for each input parameter if there are many.
- Don't skimp on explaining the details of how do you handle service errors, logic errors, system errors.
- Avoid using protocol status codes to signal service exceptions
- Avoid returning generic null type. It would have been much easier for consumer of the service if service would just return the intended type with no data i.e. <Statuses />
- Dog food your API :)
Source Code
Source code is here.
Petar