Use smileys (or emojis) with your IBM Domino REST services

I have an XPages app that has an interface to a native mobile app (running on Android and iOS). The app can be used as a "full" client to the application - so it both reads and writes data to the Domino based application.

As you probably know we have some extra keyboards on the mobile devices - which allows us to express our emotions. They are called emojis or simply smileys 😉 . Normally when we see these special characters in IBM Notes (or web based applications running on IBM Domino) they are translated to something that may not always be visible... - or more correctly, shown as a series of hieroglyphs. Now, one of our users entered a description on his cell phone and included a smiley and submitted it to the server - and it just came back "awkward" - so I decided to try and figure out how to cope with these so often-used special characters.

The journey to the result was long - and I learned a lot about unicode - and didn't understand everthing (e.g. "high surrogate" and "low surrogate"). And in the end it turned out that I didn't need to know all this!!

I use an MVC model where I read data into Java objects and after that all data handling is done through the Java objects. A Java String is naturally in unicode (actually it is the "utf-16" character set that uses two bytes per character whereas "utf-8" only uses one byte per character). In Denmark we have three extra national characters (compared to English) and for quite a while I have used "utf-8" to encode any data that needs to be sent to/read from other systems (e.g. in a REST service). This has proved to work well - that is, until now.

What I normally do is something like this:

HttpServletResponse response = (HttpServletResponse) FacesContext.getCurrentInstance().getExternalContext().getResponse();
ResponseWriter out = FacesContext.getCurrentInstance().getResponseWriter();
response.setContentType("application/json");
response.setHeader("Cache-Control", "no-cache");
response.setCharacterEncoding("utf-8");

try {

out.write(gson.toJson(data));

} catch (Exception e) {

e.printStackTrace();

} finally {

// Stop the page from further processing;
FacesContext.getCurrentInstance().responseComplete();

}

The "gson" variable is just a "GsonWrapper" as I have written about earlier. The "data" object is the Java object containing the data that I want to send out on the wire (and GSON converts that to JSON for me). The bean where this code is called is placed in an XAgent (i.e. an XPage that does not render any output):

<?xml version="1.0" encoding="UTF-8"?>
<xp:view xmlns:xp="http://www.ibm.com/xsp/core" rendered="false" viewState="nostate">
<xp:this.afterRenderResponse><![CDATA[#{javascript:ServiceController.process()}]]></xp:this.afterRenderResponse>
<xp:spanstyle="font-weight:bold">JSON Service</xp:span>
<xp:br></xp:br>
NB this page is a placeholder for returning all different data objects as JSON. Therefore, this page is NOT rendered....
<xp:br></xp:br>
</xp:view>

So far I have had the call to the processing function in the bean in the "afterRenderResponse" event of the XPage. This turns out to be important. And I cannot remember why I ended up calling it there instead of in the "beforeRenderResponse" event...

This concept has served me well - it shows all the special characters and performs well. That is until someone sent a smiley... 😀

The problem is that there are not enough bits in "utf-8" to show all of the characters in the world. However, there are som built-in magic that can use two bytes to represent a character ("utf-16" uses two bytes to show all characters - just like the good old Lotus double byte character). The beauty is that we normally don't have to care about it - the programming tools that we use will normally seamlessly handle this. However, we need to work with the "bytes" as opposed to "characters" to allow the seamless handling. So we need to send a byte stream as opposed to a String back as a response to the http request that wanted to read some data from our service. To make this happen we need to use a "DataOutputStream" instead of the "ResponseWriter" that I used above. Our code to send the response will now look like this:

HttpServletResponse response = (HttpServletResponse) FacesContext.getCurrentInstance().getExternalContext().getResponse();
response.setContentType("application/json");
response.setHeader("Cache-Control", "no-cache");
response.setCharacterEncoding("utf-8");

try {

DataOutputStream out = new DataOutputStream(response.getOutputStream());
String json = gson.toJson(result);
byte[] utf8JsonString = json.getBytes("UTF-8");
out.write(utf8JsonString, 0, utf8JsonString.length);
} catch (Exception e) {

e.printStackTrace();

} finally {

// Stop the page from further processing;
FacesContext.getCurrentInstance().responseComplete();

}

But if you had the XAgent the same way as I had then you will see an error on the server console:

HTTP JVM: java.lang.IllegalStateException: Can't get an OutputStream while a Writer is already in use

Now what is that about??? Well, if your XAgent calls the bean to handle the service in "afterRenderResponse" then a "Writer" has already been initiated. So you need to call it in "beforeRenderResponse":

<?xml version="1.0" encoding="UTF-8"?>
<xp:view xmlns:xp="http://www.ibm.com/xsp/core" rendered="false" viewState="nostate">
<xp:this.beforeRenderResponse><![CDATA[#{javascript:ServiceController.process()}]]></xp:this.beforeRenderResponse>
<xp:spanstyle="font-weight:bold">JSON Service</xp:span>
<xp:br></xp:br>
NB this page is a placeholder for returning all different data objects as JSON. Therefore, this page is NOT rendered....
<xp:br></xp:br>
</xp:view>

And that seems to be it!!! I have been over considerations about how to save data in NotesItems (as MIMEentities instead of a pure NotesItem) - and converting the emojis as per the article about MySQL (see below) - but none of this seems necessary!! - in my environment anyway.

And now our users can express emotions when describing the fish they caught (its an angler's app) - without breaking anything and without us having to remove the funny little characters!

Happy coding!!

Sources that helped me understand this problem and find the solution:

• Hacking utf16 to work around MySQL's utf8 limitations
• How to stream file from xPages?
• GSON not sending in UTF-8
• Java exception: “Can't get a Writer while an OutputStream is already in use” when running xAgent
• XPages: The Outputstream and binary data