Trading on Sentiment

How does it sound when you hear that automated algorithmic trading shall be performed based on the market sentiment analysis of your twitter updates about specific companies or stocks? Scary!!!

The potential bubble factor around such a trend is illustrated by a new hedge fund “Derwent Absolute Return” in an article by Pascal-Emmanuel Gobry in his recent article “New Hedge Fund Uses Twitter to Pick Stocks”.  Especially so when relying only on Twitter data that excludes stock specific tweets uses this information. The new fund – Derwent Absolute Return Fund from London based Derwent Capital Markets is based on a paper by researchers at Indiana University and University of Manchester. The fund has signed an exclusive deal with the researchers who published this paper, who claim that they run the twitter data through some sophisticated analysis to predict the outcome for a where the stock market was going. Here is a graphical representation of the key finding from the paper that shows the correlation between the two data series.


Figure 1 – the blue line shows the day to day DJIA values as compared to the red line that shows the Calm time series that predict the changes in the DJIA closing values that occur three days later

Figure 1 – the blue line shows the day to day DJIA values as compared to the red line that shows the Calm time series that predict the changes in the DJIA closing values that occur three days later

Or maybe it is not so far-fetched after all… that is if it is done the right way… to augment the inputs into the existing processes, data and algorithms to gain an additional insight (and not the only insight).

Today every financial institution gets market data feeds from various market data providers such as Reuters, Bloomberg, stock exchanges, etc. These data streams are collated and consolidated (these terms should ring a bell – Securities Master, Instrument Master, Master Data for Financial Instruments, Reference data) by the financial institutions and then provided to the “Quants” so that they can convert this market information into a trading strategy. However, given that every financial institution is getting the same market data, there is only so much you can do to slice and dice the same information in multiple ways. So how can these financial institutions have an upper hand over their competition? They have to look at new, timely and relevant sources of information that can help and aid in taking some informed decisions. This includes information from sources such as:

  • Twitter or other micro-blogging sites
  • Blogs where people talk about some more detailed analysis
  • News articles and media feeds to analyze market sentiment

Take a look at the following sentiment and trend information collated from a variety of sources and see if you spot any trends by simply looking at the lines and numbers in the individual charts.

Figure 2 – Google retail index (blue line) that tracks US only queries related to retail terms and so forth compared with Walmart stock price (WMT in red) movement on NYSE along with some other retailers (Target TGT in red, J C Penny JCP in green)

Figure 2 – Google retail index (blue line) that tracks US only queries related to retail terms and so forth compared with Walmart stock price (WMT in red) movement on NYSE along with some other retailers (Target TGT in red, J C Penny JCP in green)

Figure 3 – Sentiments Price correlation 30 days for Walmart from MktSentiment

Figure 3 – Sentiments Price correlation 30 days for Walmart from MktSentiment

Figure 4 – Real-time weighted Sentiment (green for positive and red for negative) for Walmart from MktSentiment

Figure 4 – Real-time weighted Sentiment (green for positive and red for negative) for Walmart from MktSentiment

Figure 5 – Negative sentiment analysis index rank from FlameIndex as compared to the stock volume for the last 30 days

Figure 5 – Negative sentiment analysis index rank from FlameIndex as compared to the stock volume for the last 30 days

Figure 6 – Mentions about Walmart across various sources such as blogs, microblogs, bookmarks, events, etc from SocialMention

Figure 6 – Mentions about Walmart across various sources such as blogs, microblogs, bookmarks, events, etc from SocialMention

If you did not see a distinct pattern that would allow you to start trading right away – then you are not the only one; I have been staring at this for a while now… 😉

While these samples are an illustration of the various pieces of information available, each one by itself is not an answer for sound predictive analytics in Capital Markets. In order for a financial institution to make sense of this data they have to put a holistic data strategy together that is able to:

  • convert this type of information into measurable metrics
  • correlate this information to the people who are influencers and the measure the potential impact of what they say
  • tie this information to the risk hierarchy of a given corporation for which the data is being analysed – to better understand the ripple effect of the exposure for a parent company
  • factor in the geo-political variations as a quantifiable metric that is tied to operations of a corporate entity in that region (wonder what will happen to all those egyptian cotton goods now…)
  • measure the impact of natural disasters on certain commodities and companies
Figure 7 – The various additional factors that will make the social data relevant to predicting the outcome of the market

Figure 7 – The various additional factors that will make the social data relevant to predicting the outcome of the market

Once this information is correlated, it needs to be converted into quantifiable and consumable upto the minute metrics that can then provide the various financial institutions with a better Social insight into the latest and greatest changes that are taking place in the Social sphere. These metrics can then be used as an addition data point for number crunching by the “Quants”.

Sounds great but how does one even start to solve this problem – the problem is of converting unstructured data into structured data, so that you can monitor and track the latest market signals based on information from the social sphere for a short peek into the near future.

Are we closer to a solution? The answer is a resounding YES. With the increased adoption of the Social Media (including social networks, social media, news articles, blogs and the new interaction models that capture your ratings, likes and dislikes), there is exabytes of unstructured data about everything, including people, places and stocks. In parallel the technology landscape has also changed in the last five years, which allows us to embark on a path to make sense and structure out of large amounts of data with cloud computing, NoSql databases, semantic analysis, and other related technologies. There is no silver bullet yet but the technologies are evolving to solve this problem.  Customers of Securities Master solutions and infrastructure (including users of MDM to consolidate financial instrument data) should pay special attention to this area as a potential extension of the capabilities that they have deployed in their enterprise.

Please feel free to share your thoughts and feedback.

Related articles:
Related companies used in the research for this article:

The new face of Enterprise Apps – Social Collaboration

The enterprise application landscape is evolving once again and the new frontier for enterprise apps is Enterprise Social Collaboration.  Historically, the enterprise apps have gone through major cycles, including waves of investments and changes in ERP, CRM, SAAS, Data Integration, MDM, BPM, Mashups, etc. However, now the trend is moving towards the use of Social Collaboration models within the enterprise. This is demonstrated by the long list of vendors (both large and small) for Enterprise Social Collaboration, as you will note from the results of Gartner Magic Quadrant 2010 for Workplace Social Software (please see the ReadWrite Enterprise article on this topic).

Short-term fad or Long-term strategy

One would argue that given the number of vendors coming into this space from different backgrounds (ranging from gaming to established enterprise vendors) and the hype of social networking sites, this is just a passing fad.  As a matter of fact, I would argue that it is the other way around because:

  1. Vendors from different backgrounds converging to the same trend validate the need from different perspective
  2. The first point also highlights the broad applicability of this type of solution
  3. Last but not the least, now is the right time for this model to work in the enterprise due to the convergence model that is emerging from other technology trends as well (described below)

All enterprise app vendors should be taking a hard look at how to incorporate such models into their application stacks. The conditions are right for the adoption of Enterprise Social Collaboration due to the convergence of the following areas in the enterprise infrastructure landscape:

  • Data Convergence – over the last few years the world of structured data has focused on improving the data backbone in an enterprise to finally breakdown and integrates the application silos at the core data level (using Data Quality and MDM). Previous attempts at doing so using EAI and ETL had failed since the focus was on messaging and data movement rather than true data integration/ data consolidation. This is one reason why you see that major data integration players now proud owners of MDM technologies (including Informatica, Tibco, IBM, SoftwareAG, etc). The strategic vision of MDM technologies has been to enable the next generation of composite applications by providing a unified data layer. While MDM has been working on the consolidation of data across structured data silos, other technologies (SOA, Mashups) have been working on the complementary aspects needed to enable such applications.
  • SOA maturity – the last few years have seen the SOA technologies and stacks become standard vocabulary for exposing and consuming interfaces. Even mainframes now have web services interfaces. REST is a step in the positive direction and making the applications interoperable.  Both MDM and SOA are an acknowledgement of the fact that the days of a wall-to-wall “single vendor” (read SAP) are long gone. The heterogeneous environments are here to stay, including integration of on-premise and off-premise applications with things as simple (or complex depends on the way you look at it) as Google Maps.
  • Mashups – did I just mention that the heterogeneous environments are here to stay… Mashups are an integral part of the application landscape, since new functionality needs to be quickly on-boarded into the enterprise without having to re-train the business users or break existing application functionality. Mashups are the way to deliver incremental functionality as widgets into the existing applications or create completely new applications. Therefore, it is no surprise that a lot of the vendors in the Enterprise Social Collaboration space come from a mashup background. If a mashup vendor has not yet claimed entry into the Enterprise Social Collaboration space then they definitely need to wake up or very soon go the way of the dodo.
  • Structured to Unstructured data – the structured and unstructured data worlds cannot stay separate for long. Every business application user needs access to related documents and content for the accounts/contacts/customers/products that they are dealing with. And every piece of unstructured content needs to be tied to a particular account/contact/customer/product or process, without which it is not of much use. That is why you see applications incorporating document/content integration capabilities and the content management suites integrating with applications. The business users do not care who provides what piece of the information; all they need is access to the data and content in a single place (regardless of how many integration points are needed by IT to make it possible). Ease of use and access to structured and unstructured data in the business context is paramount.
  • Unified communication channel – the applications for Video, Voice and Data communication are converging. You see vendors such as Microsoft getting into the voice communication business and voice vendors such as Cisco getting into data apps. The boundaries between these areas is blurring very fast, as the Business users looking for once again a seamless  and integrated experience across the multiple communication channels.
  • Search – this is another area where the difference between structured and unstructured information is becoming invisible. As and example, with the latest search interfaces and technologies, an enterprise business user has the ability to search for a customer name and not only see the results in the CRM or ERP systems but also any documents related to this customer. Yes, this does require integration, security and access but it is doable and possible today to enable this seamless user experience. The search capabilities have also incorporated “Semantic” context that makes discovery of information easier and faster for a business user.
  • Real time alerts and notifications – Notifications and Alerts have become a standard part of the enterprise vocabulary due to the capabilities provided by various applications, Business Rules Management Suites (BRMS) suites and even products such as Complex Event Processing (CEP). Any change to critical information is relayed to the users asap. Some would say that CEP was overhyped but I would disagree, I think that CEP is just finding it’s right place, not as an independent survivor but more as a complementary capability in a larger stack.
  • Crowdsourcing – for this I am going to let you look at the definition of “Crowdsourcing” where it actually originated – Wikipedia. Wikipedia has shown us how it can be put to use as a living and breathing resource of knowledge. Crowdsourcing is the next evolutionary step for Data Governance, because Data Governance in a large organization requires a specialized team (ranging in size anywhere from half a dozen to a couple of hundred people) handling the data needs for a large number of business users (ranging in size anywhere from a few hundred to a few thousand). This makes the conventional Data Governance teams more of a bottleneck. Data Governance needs to graduate from small specialized teams to the concept of crowdsourcing so that it can increase the participation of the rest of the users in the enterprise in a organic and self governing manner.
  • EMail bottleneck – “Too much email, too little time”. Email has its limitations when it comes to making the information inside emails available or consumable for knowledge sharing. This bottleneck needs to be alleviated (email will still remain the primary means of communication) by unlocking some of the information and convert it into knowledge networks. In this area, at one end companies such a Xobni have shown how client level information can be unlocked. On the other end of the spectrum you have scenarios such as advanced Email Analytics being used to drive up Marketing Campaign response rates and feed knowledge management systems.

User Focused Model

Given that some of these powerful capabilities that are already in widespread use in the enterprise landscape, what more does Enterprise Social Collaboration have to offer? The Social Collaboration software provides the “User-Centric” delivery of content/data and a seamless user experience across multiple interaction channels. Social Collaboration is a way of delivering unified content across the boundaries of structured apps, unstructured content, and communication and collaboration silos. Products in this area, such as Cisco Quad, are now not only able to deliver the data from various interaction channels such as Voice, Email and IM but also the view of the data across enterprise data apps through pluggable widgets.

The day-to-day capabilities are delivered right to the enterprise consumer of data – i.e. the Business User

The day-to-day capabilities are delivered right to the enterprise consumer of data – i.e. the Business User

Consider some of the reasons and capabilities that are going to make Social Collaboration a usable and attractive model for your enterprise users, when this Social Collaboration setting is used to deliver the data (of all types) with the user at the center of the context:

  • User Interaction model – In short, the social interaction model is “Effective and Sticky”. The effectiveness of the social interaction model has been proven by the likes of extremely popular social network companies and tools such as Facebook, LinkedIn, Twitter, etc. The enterprise application companies have always pined for ways to make enterprise software easy to use and friendly, so that the business users do not throw-up all over it. The combination of various capabilities in a Facebook style mashup seems to be working as indicated by widespread use by technical and non-technical users (full disclosure – my barely able to email mom was able to figure out how to use Facebook on her own). Enterprise users are the same people (from all different age groups) who use social networks such as Facebook to communicate with family and friends, share information and stay informed. What is the required re-training or overall training cost for something that is already widely used by everyone?
Cisco Quad user interface

Cisco Quad - Enterprise Social Collaboration user interface

  • Unified content delivery – “The composite app is dead, long live the composite application in its new incarnation”. Not only does this model have the ability to delivery timely and relevant information right into your “Inbox” but it can do so across multiple interaction channels, i.e. web and mobile. The timely and relevant information can include customer data, product information, reports, dashboards, training, documents, videos, photographs, presentations, you name it and you can fit it in this model. For Application GUI aficionados, is there a practical reason why you would not be embed parts of a CRM or ERP screens and data functionality into such a container? After all, at the end of the day, it is an example of a composite application or a portal.  Another reason why you see application behemoth SAP partnering with the likes of Jive Software or investing in efforts such as SAP Streamwork.
  • Ad-hoc process flows – The Social Interaction model facilitates ad-hoc and organic information flow without the rigid processes of a BPM implementation. If a user needs access to another expert, they only have to ask or send a request and start collaborating, without the need to define an elaborate BPM process and requesting IT for resources or help. The BPM vendors need to take note of the collaboration capabilities from a user interaction perspective. This model allows you to capture the process for the natural flow of information in an organization and then refine it; whereas the BPM tools require you to define new processes to enable collaboration without really understanding the existing flow and connections in your organization. BPM’s core strength has been process definition and orchestration; and they have always been light on GUI capabilities. Now these BPM vendors need to think about enabling ad-hoc process flows and how these ad-hoc flows can be used as a starting point for further refinement and process re-engineering.
  • Knowledge networks – Ever heard the term “Organizational Network Analysis” (ONA), if not here a nice article that illustrates the problem by describing how Influence transcends Hierarchy in any organization. To do this you need access to all the interactions that take place in an organization and the Enterprise Social Collaboration software provides such a model. That is one of the reasons why products such as Cisco Quad use an RDF store under the covers. This allows you to find the keepers of knowledge in an organization, to remove bottlenecks and improve organizational efficiencies.  The Social Interaction model allows your organization to capture key data about the connectedness of individuals in your organization, so that the pockets of knowledge can be understood and utilized. Most organizations today struggle with this problem but with no apparent solution to it. This model not only captures the interaction information (across all communication channels), but also provides the potential of analyzing this information to enable better and faster collaboration across your organization. Human Capital or Human Resource professionals should take note.
Social Networks - both internal and external to your organization

Social Networks - both internal and external to your organization with the contextual user at the center of all interactions

  • Social Analytics – Once you have understood the interaction model within your organization, the next step is to connect it external audiences and prospects. The key is to leverage relationships, both internal and external.  Understanding and leveraging such relationship networks has been a much ignored area over the last few decades but Facebook, LinkedIn and Twitter are highlighting why it is necessary to not only under the deep customer profiles but also the network of relationships that exist around a given person, once again both inside and outside your organization. This relationship network will allow organizations to understand the relationship strength within their organization and plan for potential scenarios where a rainmaker decides to leave your organization taking a whole set of clients with them (a common scenario in Financial Services and Wealth Management organizations where the Financial Advisors are known to hold on to their clients lists forever). The focus will shift from planning ahead instead of reacting later. Other potential areas for this include understanding and building stronger relationships with your customers and prospects and leveraging the influential customers to expand your business.

Your Apps and Data Strategy for Social Collaboration

The Enterprise Social Collaboration field is crowded with more than 120+ vendors selling their wares, with very few of them coming close to providing the whole range of capabilities that cover all the over scenarios. These vendors are primarily divided into three categories:

  • Social Software for Internal collaboration – these are tools that allow collaboration between your users and locations inside the organization.
  • Social Software for External collaboration – these tools allow you to communication and collaborate with external audiences such as Customers and Partners.
  • Public Social networks and Media – the externally available social networks such as Facebook, LinkedIn, Twitter etc. These are still relevant to you enterprise as they provide extensive customer and prospect information as well as insights into trends

Therefore, as the market grows and the technology matures, expect some consolidation in this area. However, if you are responsible for shaping the future of your organization then you have to look at a combination of all three models and start investing in each, with a coherent underlying strategy to connect all three categories with your existing systems in the very near future. A well thought out Enterprise Data strategy will be your key to success with Social Collaboration and delivering the promise of Social Collaboration in your enterprise environment.

MDM Adoption = engaging Business Users

In a large enterprise, often an MDM implementation is a bigger political challenge than it is a technical challenge. So at every step of the MDM journey you have to choose wisely, as it is the difference between a successful or failed MDM implementation. If you are in the middle of an MDM implementation or have just completed the first phase, then you are probably thinking what should be the next step(s):

  • Should you start looking at the daunting task of laying down Data Governance policies?
  • Tackle the beast of downstream distribution to the dozen or more downstream silos?
  • Understand the ripple effect on the integrated systems of the data consolidation that takes place in the MDM system?
  • Feed the conformed dimensions into the downstream data warehouse?

Directions to choose from after you are done with phase 1 of MDM

Choose a direction for your next MDM phase

Figure 1 – which direction to choose for increasing MDM adoption across your enterprise

Your business sponsor is probably also asking (maybe not so nicely) you when will they start to see the business value out of the multi-million dollar transformation project that you spent your last six months on.

If you want to come out of this dilemma a winner, you have to focus on the ‘Adoption of MDM” data by Business Users and establish some quick wins. You need to get the eyeballs of the business users on value of the clean and mastered data, so that they can understand, appreciate and rely on the value of the value of MDM to come back and ask for more. This way you have the business as a willing and eager participant/sponsor of your MDM master plan and help generate buy-in for the next steps of the MDM roadmap in your organization.

All this sounds great in theory but how do you actually do this in practice? … hmmm… read my bestselling book, engage me as a high paid consultant… get rid of the business users… (just kidding 😉 ).

On a serious note, how do you actually come up with a practical and convincing engagement plan? Here are 3 key pointers that will help:

  1. Listen to the business and understand their pain
  2. Listen to the business and understand their pain
  3. Listen to the business and understand their pain

The Business users are the consumers of the data in question and they interact with the data on a daily basis, through the existing functional applications such as CRM, ERP, Portals, Email clients or even smartphone extensions of your enterprise apps. They rely on the information in these systems for their day-to-day business interactions (yes, even if the data is dirty or duplicate or missing). So you have to start providing them with a comparative view of what the data looks like in the MDM system, without actually embarking on a full-blown downstream integration project. The answer is to invest in what I call the “non-Intrusive” Data Integration pattern.

Non-Intrusive Data Integration

This concept is simple and a step towards enabling MDM-aware applications in your enterprise. Here are a few steps to think about that would make your applications increasingly MDM-aware and tie MDM in the daily life of your Business Users to deliver direct and immediate value:

1. Comparative MDM views – as a first step, you need to provide your users with simple and comparative MDM views, where they can not only see the data in the native application (CRM, ERP, etc) but can also launch a URL or a window to see what additional or clean information exists in the MDM system. This does not require extensive integration but simply depends on the key identifiers in the application of choice. For example, if you are using (SFDC) as the CRM application, you only need the Salesforce ID for a customer record to query it from the MDM system using appropriate MDM APIs and enforcing the visibility rules similar to the ones in SFDC to show a small browser pop-up window or an inline-SFDC widget.  The Comparative views provide the users with the view of the MDM data in the application they use when they need it.  Even a simple view of the list of all the systems that are already tied into the same Account or Contact data (list of the cross-reference systems from the MDM hub) will go a long way in providing increased visibility to the benefit of MDM.

Custom actions in CRM/ERP applications to query and display MDM data

Custom action in CRM/ERP apps to query and display data from MDM

Figure 2 – provide business users the visibility to the comparative MDM data views from their preferred applications

2. MDM-augmented capabilities – typically CRM and ERP applications lack extensive history audit capabilities and hierarchy management capabilities as compared to MDM. MDM provides extensive capabilities for maintaining an audit trail of all the data changes; including the changes that take place through the MDM GUI and the changes that are fed in from batch processes or any upstream integration. In addition, MDM also provides the ability to collate, define and manage relationship and hierarchy details that are not typically present in functional applications such as CRM and ERP.  The view to these complementary pieces of information can once again be easily provided by creating either new pop-up widgets or custom screens in the functional application to provide the business users with the additional information that can be used either for compliance visibility (enterprise level audit trail of data changes for a given record) or deeper insights (multiple hierarchies. These capabilities for visibility to such data in functional business applications are also supported by some MDM vendors for certain applications out-of-the-box. Once again the focus is not on downstream synchronization but more on providing the non-Intrusive integration visibility for your Business users, without disrupting their day to day interactions.

Embedded MDM control in CRM/ERP apps

Embedded MDM control in CRM/ERP apps

Figure 3 – Embedded view of the MDM hierarchy in CRM

3. MDM-aware applications – the advanced piece of the non-Intrusive integration is to start making your functional applications MDM-aware by not only addressing the first two categories mentioned above, but also incrementally addressing pro-active data quality in your functional applications. Once the first phase of MDM is up and running with the core data (cleansed, matched and merged), you can start leveraging this data at the point-of-data-entry in functional applications. On data entry in functional applications, provide the Business Users with recommendations of potential matches that might already exist in another part of the enterprise (via MDM services and APIs), so that they can work with existing data instead of creating entirely new and duplicate data. This will help you move from reactive data correction to proactive data governance and start to ease the burden on your data governance team by reducing data duplication at the point-of-data-entry.

Enterprise user interaction channels

Enable MDM visibility across the different interaction channels for Enterprise users

Figure 4 – leverage the “non-Intrusive” MDM integration pattern across the various interaction channels for the Business Users in your enterprise

As you start investigating the “non-Intrusive” integration pattern for MDM, you also need to think about using it across all interaction channels (old and new) for the Business Users in your enterprise. It is an investment worth considering so that you can convert your Business Users from MDM skeptics to MDM champions.  This is especially true for new interaction channels that are not tied to the legacy infrastructure and applications, such as Mobile/Smart phones.

In summary, the path to MDM adoption requires you to implement the above creative and useful steps in order to deliver the master data to the ultimate consumers (the Business Users) in the shortest possible time with the least of amount of disruption.

My new blog…

Welcome to my new blog about Data Technologies and trends…Related Data.

If you are interested in the areas of Social Graph, Graph Analytics, Master Data Management, Enterprise Information Management, Data Integration, or the latest in the problems that Enterprise customers find difficult to solve, then you have come to the right place.