Trading on Sentiment

How does it sound when you hear that automated algorithmic trading shall be performed based on the market sentiment analysis of your twitter updates about specific companies or stocks? Scary!!!

The potential bubble factor around such a trend is illustrated by a new hedge fund “Derwent Absolute Return” in an article by Pascal-Emmanuel Gobry in his recent article “New Hedge Fund Uses Twitter to Pick Stocks”.  Especially so when relying only on Twitter data that excludes stock specific tweets uses this information. The new fund – Derwent Absolute Return Fund from London based Derwent Capital Markets is based on a paper by researchers at Indiana University and University of Manchester. The fund has signed an exclusive deal with the researchers who published this paper, who claim that they run the twitter data through some sophisticated analysis to predict the outcome for a where the stock market was going. Here is a graphical representation of the key finding from the paper that shows the correlation between the two data series.


Figure 1 – the blue line shows the day to day DJIA values as compared to the red line that shows the Calm time series that predict the changes in the DJIA closing values that occur three days later

Figure 1 – the blue line shows the day to day DJIA values as compared to the red line that shows the Calm time series that predict the changes in the DJIA closing values that occur three days later

Or maybe it is not so far-fetched after all… that is if it is done the right way… to augment the inputs into the existing processes, data and algorithms to gain an additional insight (and not the only insight).

Today every financial institution gets market data feeds from various market data providers such as Reuters, Bloomberg, stock exchanges, etc. These data streams are collated and consolidated (these terms should ring a bell – Securities Master, Instrument Master, Master Data for Financial Instruments, Reference data) by the financial institutions and then provided to the “Quants” so that they can convert this market information into a trading strategy. However, given that every financial institution is getting the same market data, there is only so much you can do to slice and dice the same information in multiple ways. So how can these financial institutions have an upper hand over their competition? They have to look at new, timely and relevant sources of information that can help and aid in taking some informed decisions. This includes information from sources such as:

  • Twitter or other micro-blogging sites
  • Blogs where people talk about some more detailed analysis
  • News articles and media feeds to analyze market sentiment

Take a look at the following sentiment and trend information collated from a variety of sources and see if you spot any trends by simply looking at the lines and numbers in the individual charts.

Figure 2 – Google retail index (blue line) that tracks US only queries related to retail terms and so forth compared with Walmart stock price (WMT in red) movement on NYSE along with some other retailers (Target TGT in red, J C Penny JCP in green)

Figure 2 – Google retail index (blue line) that tracks US only queries related to retail terms and so forth compared with Walmart stock price (WMT in red) movement on NYSE along with some other retailers (Target TGT in red, J C Penny JCP in green)

Figure 3 – Sentiments Price correlation 30 days for Walmart from MktSentiment

Figure 3 – Sentiments Price correlation 30 days for Walmart from MktSentiment

Figure 4 – Real-time weighted Sentiment (green for positive and red for negative) for Walmart from MktSentiment

Figure 4 – Real-time weighted Sentiment (green for positive and red for negative) for Walmart from MktSentiment

Figure 5 – Negative sentiment analysis index rank from FlameIndex as compared to the stock volume for the last 30 days

Figure 5 – Negative sentiment analysis index rank from FlameIndex as compared to the stock volume for the last 30 days

Figure 6 – Mentions about Walmart across various sources such as blogs, microblogs, bookmarks, events, etc from SocialMention

Figure 6 – Mentions about Walmart across various sources such as blogs, microblogs, bookmarks, events, etc from SocialMention

If you did not see a distinct pattern that would allow you to start trading right away – then you are not the only one; I have been staring at this for a while now… 😉

While these samples are an illustration of the various pieces of information available, each one by itself is not an answer for sound predictive analytics in Capital Markets. In order for a financial institution to make sense of this data they have to put a holistic data strategy together that is able to:

  • convert this type of information into measurable metrics
  • correlate this information to the people who are influencers and the measure the potential impact of what they say
  • tie this information to the risk hierarchy of a given corporation for which the data is being analysed – to better understand the ripple effect of the exposure for a parent company
  • factor in the geo-political variations as a quantifiable metric that is tied to operations of a corporate entity in that region (wonder what will happen to all those egyptian cotton goods now…)
  • measure the impact of natural disasters on certain commodities and companies
Figure 7 – The various additional factors that will make the social data relevant to predicting the outcome of the market

Figure 7 – The various additional factors that will make the social data relevant to predicting the outcome of the market

Once this information is correlated, it needs to be converted into quantifiable and consumable upto the minute metrics that can then provide the various financial institutions with a better Social insight into the latest and greatest changes that are taking place in the Social sphere. These metrics can then be used as an addition data point for number crunching by the “Quants”.

Sounds great but how does one even start to solve this problem – the problem is of converting unstructured data into structured data, so that you can monitor and track the latest market signals based on information from the social sphere for a short peek into the near future.

Are we closer to a solution? The answer is a resounding YES. With the increased adoption of the Social Media (including social networks, social media, news articles, blogs and the new interaction models that capture your ratings, likes and dislikes), there is exabytes of unstructured data about everything, including people, places and stocks. In parallel the technology landscape has also changed in the last five years, which allows us to embark on a path to make sense and structure out of large amounts of data with cloud computing, NoSql databases, semantic analysis, and other related technologies. There is no silver bullet yet but the technologies are evolving to solve this problem.  Customers of Securities Master solutions and infrastructure (including users of MDM to consolidate financial instrument data) should pay special attention to this area as a potential extension of the capabilities that they have deployed in their enterprise.

Please feel free to share your thoughts and feedback.

Related articles:
Related companies used in the research for this article:

The new face of Enterprise Apps – Social Collaboration

The enterprise application landscape is evolving once again and the new frontier for enterprise apps is Enterprise Social Collaboration.  Historically, the enterprise apps have gone through major cycles, including waves of investments and changes in ERP, CRM, SAAS, Data Integration, MDM, BPM, Mashups, etc. However, now the trend is moving towards the use of Social Collaboration models within the enterprise. This is demonstrated by the long list of vendors (both large and small) for Enterprise Social Collaboration, as you will note from the results of Gartner Magic Quadrant 2010 for Workplace Social Software (please see the ReadWrite Enterprise article on this topic).

Short-term fad or Long-term strategy

One would argue that given the number of vendors coming into this space from different backgrounds (ranging from gaming to established enterprise vendors) and the hype of social networking sites, this is just a passing fad.  As a matter of fact, I would argue that it is the other way around because:

  1. Vendors from different backgrounds converging to the same trend validate the need from different perspective
  2. The first point also highlights the broad applicability of this type of solution
  3. Last but not the least, now is the right time for this model to work in the enterprise due to the convergence model that is emerging from other technology trends as well (described below)

All enterprise app vendors should be taking a hard look at how to incorporate such models into their application stacks. The conditions are right for the adoption of Enterprise Social Collaboration due to the convergence of the following areas in the enterprise infrastructure landscape:

  • Data Convergence – over the last few years the world of structured data has focused on improving the data backbone in an enterprise to finally breakdown and integrates the application silos at the core data level (using Data Quality and MDM). Previous attempts at doing so using EAI and ETL had failed since the focus was on messaging and data movement rather than true data integration/ data consolidation. This is one reason why you see that major data integration players now proud owners of MDM technologies (including Informatica, Tibco, IBM, SoftwareAG, etc). The strategic vision of MDM technologies has been to enable the next generation of composite applications by providing a unified data layer. While MDM has been working on the consolidation of data across structured data silos, other technologies (SOA, Mashups) have been working on the complementary aspects needed to enable such applications.
  • SOA maturity – the last few years have seen the SOA technologies and stacks become standard vocabulary for exposing and consuming interfaces. Even mainframes now have web services interfaces. REST is a step in the positive direction and making the applications interoperable.  Both MDM and SOA are an acknowledgement of the fact that the days of a wall-to-wall “single vendor” (read SAP) are long gone. The heterogeneous environments are here to stay, including integration of on-premise and off-premise applications with things as simple (or complex depends on the way you look at it) as Google Maps.
  • Mashups – did I just mention that the heterogeneous environments are here to stay… Mashups are an integral part of the application landscape, since new functionality needs to be quickly on-boarded into the enterprise without having to re-train the business users or break existing application functionality. Mashups are the way to deliver incremental functionality as widgets into the existing applications or create completely new applications. Therefore, it is no surprise that a lot of the vendors in the Enterprise Social Collaboration space come from a mashup background. If a mashup vendor has not yet claimed entry into the Enterprise Social Collaboration space then they definitely need to wake up or very soon go the way of the dodo.
  • Structured to Unstructured data – the structured and unstructured data worlds cannot stay separate for long. Every business application user needs access to related documents and content for the accounts/contacts/customers/products that they are dealing with. And every piece of unstructured content needs to be tied to a particular account/contact/customer/product or process, without which it is not of much use. That is why you see applications incorporating document/content integration capabilities and the content management suites integrating with applications. The business users do not care who provides what piece of the information; all they need is access to the data and content in a single place (regardless of how many integration points are needed by IT to make it possible). Ease of use and access to structured and unstructured data in the business context is paramount.
  • Unified communication channel – the applications for Video, Voice and Data communication are converging. You see vendors such as Microsoft getting into the voice communication business and voice vendors such as Cisco getting into data apps. The boundaries between these areas is blurring very fast, as the Business users looking for once again a seamless  and integrated experience across the multiple communication channels.
  • Search – this is another area where the difference between structured and unstructured information is becoming invisible. As and example, with the latest search interfaces and technologies, an enterprise business user has the ability to search for a customer name and not only see the results in the CRM or ERP systems but also any documents related to this customer. Yes, this does require integration, security and access but it is doable and possible today to enable this seamless user experience. The search capabilities have also incorporated “Semantic” context that makes discovery of information easier and faster for a business user.
  • Real time alerts and notifications – Notifications and Alerts have become a standard part of the enterprise vocabulary due to the capabilities provided by various applications, Business Rules Management Suites (BRMS) suites and even products such as Complex Event Processing (CEP). Any change to critical information is relayed to the users asap. Some would say that CEP was overhyped but I would disagree, I think that CEP is just finding it’s right place, not as an independent survivor but more as a complementary capability in a larger stack.
  • Crowdsourcing – for this I am going to let you look at the definition of “Crowdsourcing” where it actually originated – Wikipedia. Wikipedia has shown us how it can be put to use as a living and breathing resource of knowledge. Crowdsourcing is the next evolutionary step for Data Governance, because Data Governance in a large organization requires a specialized team (ranging in size anywhere from half a dozen to a couple of hundred people) handling the data needs for a large number of business users (ranging in size anywhere from a few hundred to a few thousand). This makes the conventional Data Governance teams more of a bottleneck. Data Governance needs to graduate from small specialized teams to the concept of crowdsourcing so that it can increase the participation of the rest of the users in the enterprise in a organic and self governing manner.
  • EMail bottleneck – “Too much email, too little time”. Email has its limitations when it comes to making the information inside emails available or consumable for knowledge sharing. This bottleneck needs to be alleviated (email will still remain the primary means of communication) by unlocking some of the information and convert it into knowledge networks. In this area, at one end companies such a Xobni have shown how client level information can be unlocked. On the other end of the spectrum you have scenarios such as advanced Email Analytics being used to drive up Marketing Campaign response rates and feed knowledge management systems.

User Focused Model

Given that some of these powerful capabilities that are already in widespread use in the enterprise landscape, what more does Enterprise Social Collaboration have to offer? The Social Collaboration software provides the “User-Centric” delivery of content/data and a seamless user experience across multiple interaction channels. Social Collaboration is a way of delivering unified content across the boundaries of structured apps, unstructured content, and communication and collaboration silos. Products in this area, such as Cisco Quad, are now not only able to deliver the data from various interaction channels such as Voice, Email and IM but also the view of the data across enterprise data apps through pluggable widgets.

The day-to-day capabilities are delivered right to the enterprise consumer of data – i.e. the Business User

The day-to-day capabilities are delivered right to the enterprise consumer of data – i.e. the Business User

Consider some of the reasons and capabilities that are going to make Social Collaboration a usable and attractive model for your enterprise users, when this Social Collaboration setting is used to deliver the data (of all types) with the user at the center of the context:

  • User Interaction model – In short, the social interaction model is “Effective and Sticky”. The effectiveness of the social interaction model has been proven by the likes of extremely popular social network companies and tools such as Facebook, LinkedIn, Twitter, etc. The enterprise application companies have always pined for ways to make enterprise software easy to use and friendly, so that the business users do not throw-up all over it. The combination of various capabilities in a Facebook style mashup seems to be working as indicated by widespread use by technical and non-technical users (full disclosure – my barely able to email mom was able to figure out how to use Facebook on her own). Enterprise users are the same people (from all different age groups) who use social networks such as Facebook to communicate with family and friends, share information and stay informed. What is the required re-training or overall training cost for something that is already widely used by everyone?
Cisco Quad user interface

Cisco Quad - Enterprise Social Collaboration user interface

  • Unified content delivery – “The composite app is dead, long live the composite application in its new incarnation”. Not only does this model have the ability to delivery timely and relevant information right into your “Inbox” but it can do so across multiple interaction channels, i.e. web and mobile. The timely and relevant information can include customer data, product information, reports, dashboards, training, documents, videos, photographs, presentations, you name it and you can fit it in this model. For Application GUI aficionados, is there a practical reason why you would not be embed parts of a CRM or ERP screens and data functionality into such a container? After all, at the end of the day, it is an example of a composite application or a portal.  Another reason why you see application behemoth SAP partnering with the likes of Jive Software or investing in efforts such as SAP Streamwork.
  • Ad-hoc process flows – The Social Interaction model facilitates ad-hoc and organic information flow without the rigid processes of a BPM implementation. If a user needs access to another expert, they only have to ask or send a request and start collaborating, without the need to define an elaborate BPM process and requesting IT for resources or help. The BPM vendors need to take note of the collaboration capabilities from a user interaction perspective. This model allows you to capture the process for the natural flow of information in an organization and then refine it; whereas the BPM tools require you to define new processes to enable collaboration without really understanding the existing flow and connections in your organization. BPM’s core strength has been process definition and orchestration; and they have always been light on GUI capabilities. Now these BPM vendors need to think about enabling ad-hoc process flows and how these ad-hoc flows can be used as a starting point for further refinement and process re-engineering.
  • Knowledge networks – Ever heard the term “Organizational Network Analysis” (ONA), if not here a nice article that illustrates the problem by describing how Influence transcends Hierarchy in any organization. To do this you need access to all the interactions that take place in an organization and the Enterprise Social Collaboration software provides such a model. That is one of the reasons why products such as Cisco Quad use an RDF store under the covers. This allows you to find the keepers of knowledge in an organization, to remove bottlenecks and improve organizational efficiencies.  The Social Interaction model allows your organization to capture key data about the connectedness of individuals in your organization, so that the pockets of knowledge can be understood and utilized. Most organizations today struggle with this problem but with no apparent solution to it. This model not only captures the interaction information (across all communication channels), but also provides the potential of analyzing this information to enable better and faster collaboration across your organization. Human Capital or Human Resource professionals should take note.
Social Networks - both internal and external to your organization

Social Networks - both internal and external to your organization with the contextual user at the center of all interactions

  • Social Analytics – Once you have understood the interaction model within your organization, the next step is to connect it external audiences and prospects. The key is to leverage relationships, both internal and external.  Understanding and leveraging such relationship networks has been a much ignored area over the last few decades but Facebook, LinkedIn and Twitter are highlighting why it is necessary to not only under the deep customer profiles but also the network of relationships that exist around a given person, once again both inside and outside your organization. This relationship network will allow organizations to understand the relationship strength within their organization and plan for potential scenarios where a rainmaker decides to leave your organization taking a whole set of clients with them (a common scenario in Financial Services and Wealth Management organizations where the Financial Advisors are known to hold on to their clients lists forever). The focus will shift from planning ahead instead of reacting later. Other potential areas for this include understanding and building stronger relationships with your customers and prospects and leveraging the influential customers to expand your business.

Your Apps and Data Strategy for Social Collaboration

The Enterprise Social Collaboration field is crowded with more than 120+ vendors selling their wares, with very few of them coming close to providing the whole range of capabilities that cover all the over scenarios. These vendors are primarily divided into three categories:

  • Social Software for Internal collaboration – these are tools that allow collaboration between your users and locations inside the organization.
  • Social Software for External collaboration – these tools allow you to communication and collaborate with external audiences such as Customers and Partners.
  • Public Social networks and Media – the externally available social networks such as Facebook, LinkedIn, Twitter etc. These are still relevant to you enterprise as they provide extensive customer and prospect information as well as insights into trends

Therefore, as the market grows and the technology matures, expect some consolidation in this area. However, if you are responsible for shaping the future of your organization then you have to look at a combination of all three models and start investing in each, with a coherent underlying strategy to connect all three categories with your existing systems in the very near future. A well thought out Enterprise Data strategy will be your key to success with Social Collaboration and delivering the promise of Social Collaboration in your enterprise environment.

360° Customer view – are we there yet?

Coming from the world of Master Data Management (MDM) the “360° view of a Customer” is a much-used phrase. The concept of gaining a holistic view of the Customer has been around as long as there have been silos of data. Historically, the enterprise customers have been focused on squeezing out efficiencies out of their enterprise software environments, with:

  • Enterprise Resource Planning (ERP) – this was the first step towards standardization of enterprise systems. However, as we all know the 360° view extended beyond the ERP realm very quickly
  • Customer Relationship Management (CRM) – for a while the CRM systems promised to deliver everything for engaging a customer. Very soon, the industry realized that the customer data was spread across many silos
  • Integration Technologies, i.e. EAI/ETL – to advance the cause of a 360° view, enterprises tried to connect all the stove-pipes, which was necessary since the multiple systems were not going away but it only turned out to be temporary relief as the focus shifted to the quality of the data. EAI/ETL only delivered the transport but not the data quality and the data management processes
  • MDM/CDI (Customer Data Integration) – addressed the core problem of tying the data together across the structured data systems, both inside and outside the enterprise. However, will MDM be able to keep pace with the expanding view of the 360° view of the Customer.

The Expanding View

With the advent and proliferation of the Social Media and Social Networks, there is an added dimension of the Customer Engagement that can only be addressed by understanding the dynamic and context specific customer behavior. The marketers and the sales teams need access to the rich behavioral and sentiment data about their target customers that allows them to learn and adapt their strategy to meet the needs of todays customer. To further illustrate the adoption of Social Media the numbers from Forrester’s Groundswell content suggests that there are less than 20% of the consumers who are now staying away from Social Media technologies.

This extension of the 360° view not only applies to the Customer data but also other data domains such as Supplier, Prescriber, Vendor, Partner, Product, etc. In addition to this the other part of the extended view is the ability to understand the Influence or the Risk associated with each Customer (once again applies to other data domains as well) by understanding their Social Graph or Social Sphere.

The uncomfortable truth is that the 360° view is outpacing the tools available in the market to handle it. There is valuable data (both structured and unstructured) that is a goldmine for enterprises that can make use of it. However, this requires a structured approach and investment in building out the infrastructure and capabilities that allow you to tackle this convergence of your Enterprise data and “Big Data” from outside your firewall.

Shore up your Data Investment

To take a step in the right direction and start building for making sense of this deluge of information, you have to start investing in Data. For some enterprises MDM is the right first step, but beyond that they need to look at augmenting and adding to their customer data the various gems of information from the Social Media. This data is going to be key to how you run your next generation of campaigns, marketing, sales engagement, customer outreach, and even reacting faster to market needs and moving the needle on Innovation with a renewed customer focus. By investing in the extension of the 360° view of the Customer to the Social Media sources, you will start to evolve a structured approach to the next steps in this direction, such as:

  • Segmentation
  • Sentiment analysis
  • Trend analysis
  • Risk assessment
  • New real time metrics

The power of the Social Media is here to stay and expand. It is now time for you to expand your concept of the 360° view to move from reaction to Innovation…

Social Graph – repeats the enterprise data silo problem…

There is no doubt that social graphs are the key to enabling the next level of search and marketing (and many other use cases that I will discuss in a subsequent entry). Here are some articles that help understand the business aspect of what these social graphs or networks of interconnected nodes can deliver:

However, with the proliferation of the various Social Networking sites, each site has it’s own Social Graph. These distributed graphs are useful to whoever owns these graphs, i.e. companies such as Facebook, as it is a rich data source that can be used for marketing to target and tailor the offerings for the various users or the users these users can influence indirectly. For an organization trying to make use of these graphs, the graphs have to be combined to deliver maximum value. Without that the reach is limited to what is visible inside one network. It is hard to find a user who does not have accounts across Facebook, LinkedIn, Twitter, Flickr, etc… but it hard to find all this information in a single place.

Is this a repeat of the history? Did we not witness large organizations struggle with the mess of Data Silos inside their organizations, during the last decade? Even today it is hard to find a wall to wall SAP shop in the enterprise landscape, because one application (or in the future a Social Network) will not be the standard answer. It is safe to say that this is always going to be a world with multiple applications (within the enterprise) and multiple social networks (outside the enterprise).

At a 40000ft level, the Social Graph is a bigger data problem in the making and similar to the problem Data Silo problem that Enterprise customers faced with Enterprise apps where Customer data proliferated across various application silos, leading to integration trends such as Master Data Management.

If we are to make use of the power of Social Graphs, the dots have to be connected. At a very minimum, it requires collation and disambiguation of entities (starting with People) across various social networks so that we can know that it is the same person with multiple personalities (and avatars) across the various social networks. Only then can you start leveraging the power of the Social Graphs by freely being able to traverse the various social networks for a given profile.

Till then you can go find out what other social networks I might be a part of…

My new blog…

Welcome to my new blog about Data Technologies and trends…Related Data.

If you are interested in the areas of Social Graph, Graph Analytics, Master Data Management, Enterprise Information Management, Data Integration, or the latest in the problems that Enterprise customers find difficult to solve, then you have come to the right place.